2021MNRAS.502..662B

With the advent of deep optical-to-near-infrared extragalactic imaging on the degree scale, samples of high-redshift sources are being selected that contain both bright star-forming (SF) galaxies and faint active galactic nuclei (AGN). In this study, we investigate the transition between SF- and AGN-dominated systems at z ≃ 4 in the rest-frame ultraviolet (UV). We find a rapid transition to AGN-dominated sources brightward of M_UV ≃ -23.2. The effect is observed in the rest-frame UV morphology and size-luminosity relation, where extended clumpy systems become point-source-dominated, and also in the available spectra for the sample. These results allow us to derive the rest-frame UV luminosity function (LF) for the SF- and AGN-dominated subsamples. We find the SF-dominated LF is best fit with a double power law, with a lensed Schechter function being unable to explain the existence of extremely luminous SF galaxies at M_UV ≃ -23.5. If we identify AGN-dominated sources according to a point-source morphology criterion, we recover the relatively flat faint-end slope of the AGN LF determined in previous studies. If we instead separate the LF according to the current spectroscopic AGN fraction, we find a steeper faint-end slope of α = -1.83 ± 0.11. Using a simple model to predict the rest-frame AGN LF from the z = 4 galaxy LF, we find that the increasing impact of host galaxy light on the measured morphology of faint AGN can explain our observations.