2023MNRAS.523.2422L

Characterizing the structural properties of galaxies in high-redshift protoclusters is key to our understanding of the environmental effects on galaxy evolution in the early stages of galaxy and structure formation. In this study, we assess the structural properties of 85 and 87 Hα emission-line candidates (HAEs) in the densest regions of two massive protoclusters, BOSS1244 and BOSS1542, respectively, using the Hubble Space Telescope (HST) H-band imaging data. Our results show a true pair fraction of 22 ± 5 (33 ± 6) per cent in BOSS1244 (BOSS1542), which yields a merger rate of 0.41 ± 0.09 (0.52 ± 0.04) Gyr^–1 for massive HAEs with log (M_*/M_⊙) ≥ 10.3. This rate is 1.8 (2.8) times higher than that of the general fields at the same epoch. Our sample of HAEs exhibits half-light radii and Sérsic indices that cover a broader range than field star-forming galaxies. Additionally, about 15 per cent of the HAEs are as compact as the most massive (log (M_*/M_⊙) ≳ 11) spheroid-dominated population. These results suggest that the high galaxy density and cold dynamical state (i.e. velocity dispersion of <400 km s^–1) are key factors that drive galaxy mergers and promote structural evolution in the two protoclusters. Our findings also indicate that both the local environment (on group scales) and the global environment play essential roles in shaping galaxy morphologies in protoclusters. This is evident in the systematic differences observed in the structural properties of galaxies between BOSS1244 and BOSS1542.