The realm of the galaxy protoclusters

Abstract

The study of galaxy protoclusters is beginning to fill in unknown details of the important phase of the assembly of clusters and cluster galaxies. This review describes the current status of this field and highlights promising recent findings related to galaxy formation in the densest regions of the early universe. We discuss the main search techniques and the characteristic properties of protoclusters in observations and simulations, and show that protoclusters will have present-day masses similar to galaxy clusters when fully collapsed. We discuss the physical properties of galaxies in protoclusters, including (proto-)brightest cluster galaxies, and the forming red sequence. We highlight the fact that the most massive halos at high redshift are found in protoclusters, making these objects uniquely suited for testing important recent models of galaxy formation. We show that galaxies in protoclusters should be among the first galaxies at high redshift making the transition from a gas cooling regime dominated by cold streams to a regime dominated by hot intracluster gas, which could be tested observationally. We also discuss the possible connections between protoclusters and radio galaxies, quasars, and $\hbox {Ly}\alpha$ blobs. Because of their early formation, large spatial sizes and high total star-formation rates, protoclusters have also likely played a crucial role during the epoch of reionization, which can be tested with future experiments that will map the neutral and ionized cosmic web. Lastly, we review a number of promising observational projects that are expected to make significant impact in this growing, exciting field.

1 Introduction

1.1 Background

Galaxy clusters play an important role in numerous aspects of extragalactic astrophysics and cosmology. Because of their extreme nature, galaxy clusters and their high redshift progenitor structures featured prominently in many early discussions of cosmological models (“steady-state” versus “Big Bang”), structure formation scenarios (“top-down” versus “bottom-up”), and the properties of the dark matter (“hot” versus “cold”; e.g., Zwicky 1939; Sunyaev and Zeldovich 1972; Press and Schechter 1974; White and Rees 1978; Efstathiou and Eastwood 1981; Davis et al. 1985). The important problem of galaxy cluster formation in which small gravitational instabilities grow by many orders of magnitude in an expanding universe, first increasing in size, then contracting and finally virializing in accordance with the scaling relations of clusters also paved the way for some of the first analytical and numerical calculations of structure formation (Albada 1960, 1961; Aarseth 1963; Peebles 1970; Icke 1973). In extragalactic astrophysics, clusters are key to tracing the formation of the most massive dark matter halos, galaxies and supermassive black holes (SMBHs) (e.g., Springel et al. 2005). They are essential for galaxy formation models because of the distinct properties of galaxies in dense environments. It has been shown that in order to correctly interpret the correlations observed between galaxy properties and environment, it is important, though challenging, to separate environmental effects operating in clusters from halo assembly bias (“nurture” versus “nature”; see Lucia 2007, for a review). The latter suggests that the properties of galaxies in a halo may depend not only on the mass of that halo, but also on its formation time, and its effects may be particularly pronounced in massive halos (e.g., Gao and White 2007; Cooper et al. 2010; Wang et al. 2013; Hearin et al. 2015; Lin et al. 2016; Miyatake et al. 2016).

The gravitational collapse of galaxy clusters is best studied theoretically in numerical cosmological simulations that trace the evolution of the large-scale structure as a function of time in three dimensions. There is a vast literature on simulations of galaxy clusters that focuses primarily on clusters as a tool for constraining cosmological parameters (e.g., by comparing the number abundance, masses and density profiles of clusters at different redshifts), and on understanding the complex baryonic processes associated with massive halos. However, the very early stages of galaxy cluster formation, which takes place mostly in the linear regime prior to virialization, are not very well known. This growth phase is marked by the evolution of numerous individual galaxies forming in rather loose aggregations of dark matter halos and intergalactic gas associated with extended, overdense regions of the cosmic web at high redshift. In Fig. 1 we show an example of a galaxy (proto-)cluster in a dark matter simulation (Boylan-Kolchin et al. 2009). The different panels show the dark matter density distribution at one location at four different redshifts ($z\sim 6,2,1,0$) and on three different scales (100, 40, 15 $h^{-1}$ Mpc). A number of key characteristics of the cluster formation scenario in a ${\varLambda }$CDM universe emerge from simply looking at these pictures. The general region of a forming cluster can already be identified as early as $z\sim 6$ by means of a modest density contrast in the dark matter distribution seen on scales of several tens to a hundred Mpc. The initial matter distribution at high redshift consists of numerous smaller haloes that are clustered stronger than the dark matter, without a clear central halo, at least at $z\gtrsim 2$. This is the protocluster stage of the collapse. At later stages, a strong central mass concentration builds up, and the surrounding structure becomes more and more filamentary with large voids. At $z\sim 1$, several equally sized halos can be seen that will still merge to form the single final cluster. Many of these larger halos prior to $z=0$ are quite elongated, while the final cluster at $z=0$ is spherical. At the lower redshifts ($z\lesssim 1$) the large-scale structure surrounding the cluster appears to stabilize due to the increased expansion rate (Vikhlinin et al. 2014).

Fig. 1

The development of a massive Coma-like galaxy cluster in the Millennium II dark matter simulation. Panels show the dark matter distributions near the (proto-)cluster on 3 different scales (from left-to-right 100, 40 and 15 $h^{-1}$ Mpc) and at 4 different epochs (from top-to-bottom $z=6.20, z=2.07, z=0.99$, and $z=0$). Prior to $z\sim 2$ the main characteristic of the “protocluster” is its large-scale overdensity of dark matter (and associated gas and galaxies) that can be seen as early as $z\sim 6$ on scales of several tens of $h^{-1}$ Mpc (top row of panels). Figure reproduced from Fig. 2 in Boylan-Kolchin et al. (2009)

Figure 1 indicates that protoclusters can, in principle, be identified by their dark matter overdensities from very early on ($z\sim 6$) provided that suitable baryonic tracers exist. Their properties are important for obtaining a full census of galaxy evolution in the universe. The $\sim$4 Gyr epoch between $z\sim 4$ and $z\sim 1$ was critical for the assembly of massive galaxies. During this epoch, both the star formation and quasar (QSO) activity peaked, and most of the stellar mass in massive galaxies was assembled into individual galaxies (Madau and Dickinson 2014). Observing this crucial epoch can give very important clues to the process of galaxy formation, especially in the densest regions: these (forming) group and cluster environments contain large quantities of co-eval galaxies seen at a time when most of the “action” is taking place, magnifying details of the assembly process, including the infall of matter from the filamentary cosmic web, the fueling of star formation, interactions between galaxies, co-evolution of galaxies and their SMBHs, formation of brightest cluster galaxies (BCGs), the first stages of heating and metal-enrichment of the intracluster medium (ICM), and the build-up of the intracluster light (ICL). The highest star formation rate (SFR) occurred in the high redshift progenitors of today’s clusters, and those halos reached their peak SFRs at an earlier redshift compared to halos of lower mass (Behroozi et al. 2013). Thus, if we do not find and study these rare, massive structures at high redshift, we will lack vital empirical evidence necessary to understand cluster formation. Although some galaxy clusters with a prominent red sequence and a thermalized ICM have been identified up to $z\approx 2$, detecting their higher redshift progenitors poses serious challenges because of the subtle density contrasts involved as well as their rarity. In a sense, the study of galaxy cluster evolution significantly lags that of galaxy evolution, for which the observational frontier currently lies at $z\approx 10$ with large samples of galaxies available for statistical analysis at $z\lesssim 6$ (Bouwens et al. 2015a). Just as galaxy evolution relies on connecting present-day galaxies with their higher redshift progenitors, it is important that we study the assembly of clusters to ever more distant progenitor objects up until the smallest density fluctuations from which they originated.

1.2 Scope and outline of this review

The study of galaxy clusters goes back almost a century, and there exist many excellent reviews on galaxy clusters (e.g., Bahcall 1977; Dressler 1984; Norman 2007; Allen et al. 2011; Rosati et al. 2002; Kravtsov and Borgani 2012; Vikhlinin et al. 2014; Bykov et al. 2015). These reviews focus on the many aspects of galaxy clusters, including their use as cosmological probes, dark matter (sub-)structure, BCGs and AGN feedback, red sequence evolution, environmental effects on galaxies, ICM enrichment and hydrodynamical evolution. A thorough review of galaxy clusters is beyond the scope of this paper, and we refer the reader to the aforementioned reviews and references therein. The (protocluster) environments of high redshift radio galaxies, known to be good tracers of high-density regions, were reviewed as part of Miley and Breuck (2008), and many other recent papers referenced in this review present detailed views of many aspects of protoclusters. In this paper, we compile an introductory level review of this relatively young but rapidly growing field. The structure of this review is as follows. In Sect. 1.3, we will give some necessary definitions related to the term “protocluster”. We will start by giving a brief overview of recent high redshift cluster surveys (Sect. 2.1), followed by a description of the main techniques used for finding protoclusters (Sect. 2.2). In Sect. 3, we present the existing sample of protoclusters selected from the literature and discuss the main observational characteristics such as redshifts, overdensities, and derived masses. The properties of protoclusters are described in more detail in Sect. 4, with an emphasis on the dark matter and gas accretion history of protoclusters (Sect. 4.1), development of the cluster red sequence (Sect. 4.2), BCGs (Sect. 4.3), and the comparison between galaxies in protoclusters and in the field (Sect. 4.4). Next, we will take a closer look at interesting possible connections between protoclusters and $\hbox {Ly}\alpha$ blobs (Sect. 5.1) and radio galaxies and quasars (Sect. 5.2). We will discuss the role of protoclusters during the epoch of reionization in Sect. 5.3. We conclude this review with a brief overview of new surveys and instruments that may have a high impact on this emerging field (Sect. 6.1), and by highlighting a number of key open topics (Sect. 6.2).

1.3 Definition of clusters and protoclusters

The term “cluster” is used differently by different authors. Many authors reserve this term for the most massive, virialized systems ($M\gtrsim 10^{14} M_\odot$), while referring to systems with masses between ${\sim }10^{13} M_\odot$ and ${\sim }10^{14} M_\odot$ as “groups”. However, for lack of a real physical distinction between the two, other studies make no distinction and refer to all massive systems above some threshold mass collectively as “groups” or “clusters”. In this review, we will follow the popular convention that clusters are virialized objects with total masses $\gtrsim 10^{14} M_\odot$ (e.g., see Bower and Balogh 2004). Because the halo mass function is very steep at high masses, the distinction between groups ($10^{13}<M/M_\odot <10^{14}$) and clusters ($M/M_\odot \ge 10^{14}$) helps to more easily track their evolution when comparing to theoretical models or cosmological simulations.

The study of protoclusters has been somewhat plagued by the fact that the term “protocluster” is also used differently by different authors and different studies. However, if we adopt the simple definition of a “cluster” given above, the meaning of the term “protocluster” becomes straightforward: a protocluster is a structure that will, at some stage, collapse into a galaxy cluster (i.e., a virialized object of $\ge 10^{14} M_\odot$ at $z\ge 0$). This definition has the convenient implication that the combined space number density of clusters and protoclusters at any redshift is equal to the abundance of clusters today¹.

It is important to keep in mind that even when we provide a strict definition of what constitutes a protocluster, it is really only practical in theory and simulations, and not in observations. The observational data are often insufficient to decide with absolute certainty if we are dealing with the progenitor of a galaxy cluster. Moreover, in order to be able to classify an object as a protocluster, we require not only detailed knowledge about the distant object itself, but also about its descendant at the present-day. Especially the latter requirement presents a serious challenge that typically can only be addressed in a statistical way rather than on an object-by-object basis. Note that for the lower redshift galaxy clusters this is typically not a problem, as the presence of the cluster red sequence, the velocity dispersion of the galaxies, or the X-ray, Sunyaev–Zel’dovich (SZ) effect or gravitational lensing mass profiles usually provide accurate enough constraints on the system’s mass, and its dynamical and evolutionary state.

In practice, many authors therefore use a somewhat stricter definition of the term protocluster as being a structure that is sufficiently overdense compared to its surroundings such that it can be recognized observationally. This does not conflict very strongly with the all-encompassing definition of all cluster progenitors that we use here. However, it is important to realize that for every protocluster observed, there are typically many other that have not yet developed a significant density contrast or that are missed in observations because of incompleteness or sensitivity. Another subtlety arises when we consider that many virialized clusters are still surrounded by other material that has yet to become part of the cluster through infall or merging (Chiang et al. 2013; Muldrew et al. 2015). In these cases, the protocluster could be seen as a much larger region that includes the entire structure. However, we will follow the current literature convention to call any collapsed object of at least $10^{14} M_\odot$ a cluster, and call it a protocluster when such a massive core is not yet present. However, it is important to remember that cluster formation is a process that is occurring at all redshifts. An example of a protocluster at the relatively low redshift of $z\sim 0.4$ is the super-group SG1120–1202 (Gonzalez et al. 2005; Kautsch et al. 2008; Smit et al. 2015), which is expected to collapse to form a Virgo-sized cluster by the present-day. In this review, however, we will focus on protoclusters at much higher redshifts. This choice is not entirely arbitrary as the redshift range $1.5\lesssim z\lesssim 2.5$ covers the transition epoch for typical massive clusters and their protocluster progenitors.

2 Searching for protoclusters

2.1 High redshift cluster surveys

There is a long and rich tradition of surveys aimed at detecting galaxy clusters at increasingly high redshifts. The main searches are centered around looking for concentrations of red sequence and other massive galaxies, or looking for the characteristic signature of hot cluster gas either in the X-rays or using the SZ effect. The frontier of cluster searches currently lies at $z\simeq 1.5-2$, which has been identified as a very important epoch for massive galaxy clusters.

The pioneering work on the Red Sequence Cluster Survey (RCS) by Gladders and Yee (2000, 2005), and its successor, RCS-2 (Gilbank et al. 2011), showed that large samples of clusters can be efficiently selected out to $z\approx 1$ by making use of standard optical filters combined with the z-band to pinpoint overdensities of galaxies as their Balmer or 4000 Å break redshifts through the optical and NIR filters. By extending the red sequence search strategy to near- and mid-IR filters (e.g., using the colors $J-K$ or $z^\prime -[3.6~\upmu \hbox {m}$]), the red sequence technique has been successfully employed at $z>1$, for example using the UKIRT Infrared Deep Sky Survey (UKIDDS; Andreon et al. 2009) and the Spitzer Adaptation of the RCS (SpARCS; Muzzin et al. 2009; Wilson et al. 2009). Although the $z^\prime -[3.6~\upmu \hbox {m}$] color nicely straddles the red sequence at these high redshifts, galaxies are about 2 magnitudes fainter at $z\sim 2$ with respect to $z\sim 1$, requiring imaging depths that pose a significant challenge to finding more distant clusters (Muzzin et al. 2013). One solution to this problem was obtained by Papovich (2008). To identify overdensities of co-eval galaxies, they used the redshifted 1.6 $\upmu$m ‘stellar bump’ feature, which is ubiquitous in the spectra of most types of galaxies. At $z\gtrsim 1$, the color [3.6 $\upmu$m] – [4.5 $\upmu \hbox {m}$] increases monotonically with redshift up to $z\sim 1.7$, after which it reaches a plateau. Foreground galaxies can furthermore be removed by involving an optical or near-IR filter (Papovich 2008; Falder et al. 2011; Muzzin et al. 2013). In order to construct a high redshift sample of cluster candidates, Papovich (2008) selected the $3\sigma$ overdensities from the density map of the stellar bump galaxies smoothed on a scale corresponding to the expected size of a cluster core. This resulted in a large sample of candidate (proto-)clusters with the number density and clustering expected of the progenitors of present-day clusters. One of the candidates was spectroscopically confirmed to be at $z=1.62$ (Papovich et al. 2010), while another $z=2.00$ cluster² having a strong red sequence of early-type galaxies was also identified using this technique (Gobat et al. 2011; Strazzullo et al. 2013). The Clusters Around Radio-Loud AGN (CARLA) survey targeted several hundred radio-loud quasars and radio galaxies at $z>1.3$ with the Spitzer Space Telescope, discovering several new mid-IR selected clusters (Galametz et al. 2012; Wylezalek et al. 2013; Cooke et al. 2015, 2016; Noirot et al. 2016, see also Sect. 5.2.1). Using both the stellar bump method and the red sequence technique, Muzzin et al. (2013) have detected a rich cluster at $z=1.63$, while Rettura et al. (2014) found a large number of candidate clusters in the redshift range $1.3<z<2.0$ in the 100 $\hbox {deg}^{2}$ Spitzer South Pole Telescope Deep Field (SSDF; Ashby et al. 2013).

Instead of color selections, that may rely on the type of stellar populations present in cluster galaxies as a function of redshift, a more complete selection of distant clusters can be made by searching for overdensities in redshift space, provided that accurate photometric redshifts over a large survey area are available (e.g., Breukelen and Clewley 2009; Eisenhardt et al. 2008; Chiaberge et al. 2010; Spitler et al. 2012; Castignani et al. 2014; Chiang et al. 2014; Ascaso et al. 2015; Wang et al. 2016). Strazzullo et al. (2015) searched for overdensities of passive galaxies with photometric redshifts using nearest neighbor or fixed physical radius densities, finding several promising candidates in the range $1.5< z < 2.5$. The IRAC Shallow Cluster Survey (ISCS; Eisenhardt et al. 2008) and its higher redshift extension, the IRAC Distant Cluster Survey (IDCS; Stanford et al. 2012), searched for redshift overdensities to identify a large sample of $z>1$ cluster candidates of which many have since been confirmed, including the extremely massive cluster IDCS J1426+3508 at $z=1.75$ (e.g., Brodwin et al. 2011, 2016; Stanford et al. 2012). The cluster JKCS 041 at $z=1.80$ was discovered using the red sequence technique (Andreon et al. 2009). It has a large mass (${>}10^{14} M_\odot$) based on both the mass-richness relation and the detection of a hot ICM (Andreon et al. 2014).

Focusing instead on their gaseous components as a means of discovering high redshift clusters, existing wide-field X-ray surveys are at the limits of their sensitivity for clusters at about $z\sim 1.5$ (Rosati et al. 2002; Mullis et al. 2005; Stanford et al. 2006). Nonetheless, a recent deep 50 deg$^2$ cluster survey with XMM-Newton has led to the discovery of one of the highest redshift clusters, XLSSU J021744.1-034536 at $z\simeq 1.91$, which was also detected using SZ effect follow-up observations (Mantz et al. 2014). SZ surveys are becoming more and more powerful for identifying high redshift clusters through the spectral distortion of the cosmic microwave background as its photons traverse the ICM. Bleem et al. (2015) found hundreds of new clusters in the 2500 $\hbox {deg}^2$ South Pole Telescope SZ survey (SPT-SZ). SPT-SZ clusters have a median redshift of $z\sim 0.55$ with a significant tail extending to much higher redshifts, as evidenced by the discovery of SPT-CL J2040-4451 at $z=1.478$. Although the detection rate of very high redshift clusters using blind SZ effect and X-ray surveys is still lagging behind that of galaxy-based surveys, this situation may soon change due to upgraded SZ instrumentation at the SPT and the Atacama Cosmology Telescope, as well as new X-ray missions.

2.2 Protocluster searches at $z\gtrsim 2$

As we move into the epoch during which the galaxies that are in (proto-)clusters become difficult to distinguish from galaxies that are in the field due to the absence of a well-defined red sequence or a hot ICM, the most conspicuous sign that indicates the presence of a (proto-)cluster is the relatively large concentration of galaxies and associated gas clustered in angular coordinates and redshift space. The main techniques for finding protoclusters are therefore constructed around this simple observational signature. Below we will review the main methods that are being used to find protoclusters.

2.2.1 Surveys and “serendipitous” discoveries

Because galaxy clusters are a fundamental component of the large-scale structure of the present-day universe as they occupy the dense nodes in the filamentary cosmic web, surveys capable of tracing the formation of this cosmic web at high redshift are, in principle, also surveys of cluster formation. In practice, the study of cluster formation with standard galaxy surveys is extremely complicated through a number of reasons. Clusters are rare objects in the local universe, and extremely large cosmic volumes need to be surveyed to find their progenitors. The density contrasts between (proto-)clusters and the field are relatively small at high redshift, requiring sensitive surveys and, ideally, good spectroscopic coverage in order to confirm candidates and disentangle the real structures from the projection effects.

Several protoclusters have already been found as by-products of large spectroscopic surveys. Good examples of such discoveries are the protoclusters at $z=2.30$ and $z=3.09$ discovered in the HS1700+643 and SSA 22 fields, respectively (Steidel et al. 1998, 2000, 2005). Both objects were found through rest-frame UV spectroscopy of candidate high redshift galaxies selected on the basis of the Lyman break. At present, the size of the cosmological volumes at high redshift that can be surveyed using this technique are still relatively small. However, it is possible to speed up this cumbersome process by pre-selecting the most overdense regions (in projection) from wide-field photometric surveys of, e.g., Lyman break galaxies or Ly$\alpha$ emitters, and following up spectroscopically only the most overdense regions. Good examples of early structures hence identified are the large structures at $z=4.86$ and $z=6.01$ both in the Subaru Deep Field (Shimasaku et al. 2003; Toshikawa et al. 2012), and at $z=5.7$ in the Subaru/XMM-Newton Deep Field (Ouchi et al. 2005). Recently, Toshikawa et al. (2016) performed a protocluster survey in the Canada–France–Hawaii Telescope Legacy Survey (CFHTLS) deep fields covering an area of 4 deg$^2$ with sufficient filters and depth to make surface density maps of dropout galaxies at $z\simeq 3-6$. They followed up spectroscopically only the most significant ($>4\sigma$) overdensities for which simulations predict that ${\sim }$80% are genuine protoclusters, finding several new systems. Chiang et al. (2014) used a catalog of accurate photometric redshifts in the $1.6\,\hbox {deg}^2$ COSMOS field to select overdense regions in three-dimensional space (sky position and redshift) that are good candidates of Coma-like protoclusters at $z\simeq 2-3$. They found 36 candidates, several of which have since been confirmed to be genuine protoclusters (e.g., Chiang et al. 2015; Diener et al. 2015; Lee et al. 2014b; Mukae et al. 2016; Wang et al. 2016). The two-step approach of first identifying surface overdensities, possibly aided by photometric redshifts, in a wider field survey followed by spectroscopy is significantly faster than performing a spectroscopically complete survey, although it has the disadvantage that there is no complete spectroscopic sampling of the cosmic web and one is most sensitive to structures that are relatively compact in the sky plane.

The VIMOS Ultra Deep Survey (VUDS; Fèvre et al. 2015) is currently the largest spectroscopic survey (in terms of spectra per square degree) capable of sampling the cosmic web at high redshift ($z\simeq 2-6$) with a sampling that is sufficiently dense and uniform to uncover a wide range of structures including protoclusters. The survey has already resulted in two exceptionally overdense, massive structures at $z=2.9$ and $z=3.3$ (Cucciati et al. 2014; Lemaux et al. 2014), and several tens more are expected to follow after analysis of the full VUDS data set.

A large number of protoclusters were discovered serendipitously using data from the Planck survey. The Planck all-sky (sub-)millimeter maps contain a large number of compact (unresolved) “cold” sources (Clements et al. 2014; Planck Collaboration et al. 2015). A large fraction of these sources are believed to be overdensities of star-forming galaxies in overdense regions at $z\simeq 2-4$ with significant far-infrared emission redshifted into the 353–857 GHz frequency range (Planck Collaboration et al. 2015). The Planck selection resulted in over one thousand candidates over 26% of the sky, of which about two hundred were followed up with Herschel to obtain higher resolution data. Flores-Cacho et al. (2016) showed that at least one of the structures is due to two overlapping high redshift structures along the line of sight, at $z\sim 1.7$ and $z\sim 2$. Although most of the Planck sources remain to be confirmed spectroscopically, the uniform selection method and the large survey area make this a highly promising new search technique. The Planck selection is also interesting because it complements other protocluster selections that are based on optical- and NIR-selected galaxies (relatively dust-free, star-forming galaxies and passive galaxies).

2.2.2 Biased tracer techniques

Finding galaxy (proto-)clusters at high redshift is like searching for a needle in a haystack. The biased tracer technique attempts to bypass the complications of carrying out a very deep and wide galaxy survey by directly targeting the immediate environment of previously identified galaxies that are known to be good tracers of massive forming systems. Early protocluster studies have therefore targeted objects for which there is strong evidence that they are progenitors of the very massive galaxies that populate the centers of today’s clusters.

One such class is that of the high redshift radio galaxies that have many properties indicative of progenitors of local BCGs (e.g., Best et al. 1998; Zirm et al. 2003; Miley and Breuck 2008; Collet et al. 2015). A substantial number of protoclusters were found by targeting the environment of previously known radio galaxies (e.g., Pascarelle et al. 1996; Fevre et al. 1996; Pentericci et al. 2000; Kurk et al. 2000, 2004a; Venemans et al. 2002, 2004, 2005, 2007; Chiaberge et al. 2010; Kuiper et al. 2011; Hatch et al. 2011a, b; Mawatari et al. 2012; Hayashi et al. 2012; Wylezalek et al. 2013; Cooke et al. 2014). In many of these studies, the first step was to use a narrow-band filter to identify candidate emission line galaxies such as $\hbox {Ly}\alpha$ emitters (LAEs) or $\hbox {H}\alpha$ emitters (HAEs) near the redshift of the radio source, often followed up by spectroscopy in order to confirm the candidates. By comparing the surface or volume densities of the emission line candidates near the radio sources with those obtained from field surveys, a substantial number of radio sources at high redshift was found to be associated with overdensities that were interpreted as protoclusters. The overdense fields were subsequently followed up at other wavelengths, showing in many cases that the overdensities are accompanied by overdensities of other galaxy populations as well, including red (sequence) galaxies (e.g., Kurk et al. 2004b; Kodama et al. 2007; Zirm et al. 2008), HAEs (Kurk et al. 2004a; Hatch et al. 2011b; Shimakawa et al. 2014; Cooke et al. 2014), Lyman Break Galaxies (LBGs) (e.g., Intema et al. 2006; Overzier et al. 2006, 2008; Capak et al. 2011) or Sub-mm Galaxies (SMGs) (e.g., Breuck et al. 2004; Dannerbauer et al. 2014; Rigby et al. 2014).

Like the radio galaxies, the population of QSOs at high redshift is also frequently associated with massive galaxy and galaxy cluster formation. Many studies have targeted QSOs at $z\simeq 2-6$ in order to study their immediate environments and look for protoclusters, sometimes finding overdensities and sometimes finding no difference with the random expectation (e.g., Djorgovski et al. 2003; Wold et al. 2003; Kashikawa et al. 2007; Stevens et al. 2010; Falder et al. 2011; Matsuda et al. 2011; Trainor and Steidel 2012; Husband et al. 2013; Adams et al. 2015; Hennawi et al. 2015). Much of the observational work has concentrated on the environment of the highest redshift QSOs at $z\gtrsim 6$ (e.g., Stiavelli et al. 2005; Kim et al. 2009; Overzier et al. 2009a; Bañados et al. 2013; Morselli et al. 2014; Simpson et al. 2014), because simple theoretical arguments and some simulations suggest that these should be clear cases of highly overdense regions (e.g., Springel et al. 2005). This is the topic of Sect. 5.2 of this review.

Other sources that have been suggested to be good tracers of overdense regions at high redshift are Ly$\alpha$ blobs (LABs) and SMGs. LABs are believed to trace large reservoirs of cool intergalactic gas associated with dense locations in the cosmic web, where it is being ionized and illuminated by a powerful AGN or starbursts (see Sect. 5.1). Narrow-band observations of protoclusters have shown that these LABs are preferentially found in overdense regions at high redshift, motivating the search for protoclusters near LABs as well (Prescott et al. 2008, 2012). Dusty star-forming galaxies are believed to be the progenitors of present-day massive elliptical galaxies. Overdensities of these galaxies at high redshift could thus indicate a protocluster seen at an early stage when a large number of proto-ellipticals were going through their main star formation phase. This has motivated many studies to target the dusty galaxy population in protoclusters (e.g., Stevens et al. 2003; Greve et al. 2007; Priddey et al. 2008; Daddi et al. 2009; Carrera et al. 2011; Ivison et al. 2013; Casey et al. 2015; Planck Collaboration et al. 2015; Valtchanov et al. 2013; Dannerbauer et al. 2014; Rigby et al. 2014; Umehata et al. 2014, 2015).

2.2.3 Gas absorption studies: a promising new technique

A relatively new technique for the selection of protoclusters is to make use of the fact that overdense regions in the early universe are not only overdense in dark matter and galaxies, but should also contain large quantities of cold or warm, dense gas that can be detected in absorption against a suitable background continuum source such as QSOs and galaxies. Several protoclusters have been detected through neutral hydrogen absorption along the line of sight to background quasars (e.g., Francis et al. 1996; Steidel et al. 1998; Hennawi et al. 2015). Other structures have been detected by stacking the spectra of background galaxies, and searching for Ly$\alpha$ or Mg II $\lambda \lambda$2797,2803 absorption at the redshift of the overdensity (e.g., Giavalisco et al. 2011; Cucciati et al. 2014).

Recent authors have therefore suggested to search for the presence of protoclusters through absorption in a more systematic way. By taking spectra of a background population (star-forming galaxies and QSOs) that is sufficiently densely spaced on the sky, it will be possible to perform “tomography” of the cosmic web simply by mapping out the Ly$\alpha$ absorption caused by the neutral hydrogen in the IGM (Stark et al. 2015; Lee et al. 2014a; Cai et al. 2015). Initially, authors suggested to use sightlines to background QSOs drawn from large spectroscopic surveys (e.g., Caucci et al. 2008). Even when the sightlines to background objects are spaced relatively sparsely, a larger than average optical depth that is coherent on large scales along the line of sight can be used as a tracer for highly overdense regions. Frye et al. (2008) and Matsuda et al. (2010) identified a ${\sim }80$ Mpc overdense region through the imprint of many overlapping absorbing HI clouds seen in the spectrum of a galaxy at $z=4.9$. Finley et al. (2014) serendipitously found a number of highly clustered regions of absorbing gas at $z=2.7$ along the sightlines to two background QSOs separated by about 100 kpc on the sky. The absorbers extend over about 2000 km s$^{-1}$ or 6.4 $h_{70}^{-1}$ Mpc, and may be due either to a protocluster or a filament seen along the line of sight. Cai et al. (2015) used simulations to study the correlation between the optical depth of the IGM and mass on scales of (15 $h^{-1}$ Mpc)$^3$ (i.e., protocluster scale regions) at $z=2.5$, finding that the highest effective optical depth regions trace the most massive structures. Although some signal confusion is expected from individual Damped Lyman-$\alpha$ Absorbers (DLAs) and clustered Lyman Limit Systems (LLSs), they show that they can pinpoint several previously known and unknown overdensities by means of this technique.

Fig. 2

Mock absorption maps for tomographic mapping of the IGM as predicted for 5 massive structures at $z=2.5$. Panels show the ideal and reconstructed maps for different average transverse separations of the background sources ranging from 2–6 Mpc (left-to-right), and for various scenarios (top-to-bottom, see Stark et al. 2015, for details). Figure reproduced from Fig. 8 in Stark et al. (2015)

At magnitudes of $g\gtrsim 24$ the number density of star-forming galaxies (LBGs) exceeds that of the QSOs by more than a factor of 10 (e.g., see Figure 1 in Lee et al. 2014a), allowing tomographic IGM maps to be constructed with a much higher spatial resolution. Until recently, large spectroscopic surveys reaching these magnitudes over large cosmological volumes were unfeasible, but this situation is changing. The observational requirements for performing such a survey at $z\approx 2$ were investigated by Lee et al. (2014a) and Stark et al. (2015). Stark et al. (2015) looked at the specific absorption signatures of protoclusters using mock maps expected for a range of typical separations of background sightlines. A figure from that study is presented in Fig. 2, which shows the mock absorption maps predicted for 5 massive structures at $z=2.5$ and the likelihood of identifying such structures correctly. Lee et al. (2014a) furthermore showed that with limiting g-band magnitudes of 24.0, 24.5, and 25.0 (corresponding to sightline densities of ${\sim }$360, 1200, and 3300 galaxies $\hbox {deg}^{-2}$), a three-dimensional map of the IGM can be reconstructed having a spatial resolution of only a few Mpc. They further show that the experiment could be performed with moderate resolution (${\simeq }1000$) multiplexed spectrographs on 8–10 m class telescopes covering an area of 1 deg$^2$ (volume of order $10^6 h^{-3}$ Mpc$^3$) in 100 h. A Ly$\alpha$ tomographic survey is more efficient for mapping the three-dimensional cosmic web than a galaxy redshift survey at the same redshift. If large cosmological volumes can be mapped, the Ly$\alpha$ tomographic mapping technique would be one of the most powerful probes for uncovering for the first time very large, unbiased samples of protoclusters. In order to demonstrate the technique on real data, Lee et al. (2014b) applied the Ly$\alpha$ tomographic mapping technique in a $5'\times 14'$ region of the COSMOS field. By targeting a small number of star-forming galaxies at $z>2.3$, a map of the large-scale structure as traced by Ly$\alpha$ absorption was made over a $6\times 14\times 240$ Mpc region at a spatial resolution of ${\approx }3.5 h^{-1}$ Mpc. The map resolution was sufficient to show that galaxies follow several high-density peaks in the absorbing hydrogen distribution. Interestingly, the highest density feature in the absorption map of Lee et al. (2014b) lies along the line of sight toward a large-scale overdensity of LAEs at $z=2.44$ found independently by Chiang et al. (2015), showing the potential for discovering protoclusters using the Ly$\alpha$ tomography technique.

3 The protocluster “zoo”

3.1 Overview of objects found to date

The techniques described in the previous section have resulted in a wide range of interesting structures that all highlight different aspects of cluster formation. However, taken together it is a highly heterogeneous data set, and it is currently still difficult to paint a complete picture of cluster evolution using these data. So what has been found so far? Some illustrative examples of the sky and redshift distributions of galaxies in protoclusters are shown in Figs. 3 and 4. These protoclusters were identified based on their large-scale (several to several tens of Mpc comoving) overdensities of various galaxy populations (e.g., star-forming galaxies, LBGs, HAEs and LAEs) that are strongly clustered in projected sky coordinates (Fig. 3) or in redshift space (Fig. 4). In Fig. 5, we present the redshift distribution of all protoclusters identified to date. We have based our compilation on the objects listed in Table 5 of Chiang et al. (2013), supplemented by new objects discovered since (Cucciati et al. 2014; Lemaux et al. 2014; Lee et al. 2014; Diener et al. 2015; Chiang et al. 2015; Casey et al. 2015; Toshikawa 2014; Toshikawa et al. 2016; Ishigaki et al. 2016; Wang et al. 2016). We have included all structures at $z\gtrsim 1.5$ having estimates of their present-day masses in excess of ${\sim }10^{14} M_\odot$. About half of all known protoclusters are found in the ${\sim }1$ Gyr epoch between $z=2$ and $z=3$, while the other half is found in the ${\sim }1.5$ Gyr epoch between $z=3$ and $z=8$. Note, however, that we have not included here candidates resulting from “snap-shot” type and statistical surveys that have uncovered large numbers of as of yet unconfirmed candidates (e.g., Wylezalek et al. 2013; Planck Collaboration et al. 2015; Chiang et al. 2014; Franck and McGaugh 2016). The total number of potential protoclusters found is therefore probably substantially larger than that shown in Fig. 5, as discussed elsewhere in this review.

Fig. 3

Examples of protoclusters, highlighting the spatial distributions of member galaxies in these overdense structures. The left panel shows the distribution of galaxies with photometric and spectroscopic redshifts in a protocluster at $z\approx 1.62$ [red squares red galaxies satisfying $z-J>1.3$; blue circles galaxies satisfying $\hbox {SFR}>5 M_\odot \hbox {year}^{-1}$; black circles other galaxies; Hatch et al. (2016)]. The middle panel shows a protocluster at $z\approx 2.16$ (red, green and blue squares HAEs with relatively red, intermediate and blue colors; black triangles HAEs not detected in the K-band; Koyama et al. (2013a)). The right panel shows a $z=3.78$ protocluster in the Bootes field (green squares LAEs at $3.77<z<3.81$; black circles other LAEs; Dey et al. (2016))

Fig. 4

Examples of protoclusters at $z\sim 2-6$, highlighting the relatively narrow redshift distributions of spectroscopically confirmed galaxies in these overdense structures. References: Diener et al. (2015, D15), Mostardi et al. (2013, M13), Venemans et al. (2007, V07), Steidel et al. (2003, S03), Venemans et al. (2005, V05), Lemaux et al. (2014, L14a), Lee et al. (2014, L14b), Dey et al. (2016, D16) and Toshikawa (2014, T14)

Fig. 5

The distribution of redshifts of protoclusters selected from the literature (large blue symbols). The data for clusters below $z=1.5$ were taken from the compilation of clusters detected in X-ray and SZ surveys of Bleem et al. (2015). Large red symbols are high redshift clusters at $z>1.6$. The position of objects along the polar axis holds no information and is used for visibility purposes only. Redshifts and ages (in Gyr) are indicated along the radial axes

Figure 6 shows the range of galaxy overdensities ($\delta _\mathrm{gal}$) that have been measured for a subset of the protoclusters shown in Fig. 5. For each study, we have indicated the galaxy type that was used to measure the overdensities (LAEs, HAEs, LBGs or other), and whether the protocluster was found in the field or using a tracer (radio galaxy, QSO or other). These results show that although there already exists a relatively rich data set of all kinds of cluster progenitors, these data are rather heterogeneously selected and cover a wide redshift range. The measurements collected in this figure only serve to illustrate the range of overdensities typically associated with protoclusters, and should generally not be compared directly, either in the absolute or in the relative sense without taking into account the different selection and measurement techniques used by each study. Nonetheless, the range of overdensities measured for LAEs, LBGs and HAEs at these redshifts are broadly consistent with the overdensities expected for galaxy protoclusters based on the simulations of Chiang et al. (2013), who calculated the overdensities of galaxies with SFRs $>1 M_\odot \,\hbox {year}^{-1}$ in a volume of (15 $h^{-1}\, \hbox {Mpc})^3$ around protoclusters.

Fig. 6

The diversity of overdensities of galaxies measured for a selection of protoclusters at $z>2$. Symbols indicate protoclusters found using radio galaxies (circles), quasars (stars), other tracers (pentagons), and in the field (squares). Colors indicate overdensity values measured using LAEs (green), LBGs (blue), HAEs (red) or other types of galaxies (black). The measurements collected in this figure only serve to illustrate the range of overdensities typically associated with protoclusters, and should generally not be compared directly, either in the absolute or in the relative sense without taking into account the different selection and measurement techniques used by each study

3.2 Why we believe these are protoclusters

The structures shown in Figs. 3, 4, 5, 6 span a rather large range in redshift, (co-moving) size, (projected) shape, and galaxy overdensity. In Fig. 7, we show the distribution of the present-day masses estimated for the protoclusters selected from the literature. The median protocluster has $\mathrm {log}(M/M_\odot )_{z=0} = 14.6$, and the sample includes some exceptional protoclusters with ($z=0$) masses of $\gtrsim 10^{15} M_\odot$ (e.g., Venemans et al. 2002; Steidel et al. 2005; Cucciati et al. 2014; Lemaux et al. 2014; Diener et al. 2015). These present-day masses of protoclusters span the full range of masses of typical galaxy clusters found in X-ray and SZ effect surveys (see Bleem et al. 2015, for a recent compilation), suggesting that the term “protocluster” is justified.

Fig. 7

The distribution of present-day ($z=0$) masses derived for the protocluster sample from Fig. 5 (black histogram), compared to the mass distribution of galaxy clusters discovered as part of X-ray (red histogram), South Pole Telescope (SPT, green histogram), and Planck (blue histogram) surveys (Bleem et al. 2015)

How were these mass estimates obtained? In order to show that some overdense region of galaxies identified at high redshift will evolve into a present-day cluster, at least two requirements need to be fulfilled. First, the overdensity of the region needs to be large enough to enable it to detach from the expanding part of the universe around it, contract, and collapse. To guarantee that the region will be a galaxy cluster, the total mass of the final collapsed region needs to be at least $10^{14} M_\odot$. Second, because we are interested in the progenitors of present-day galaxy clusters, this process must be completed before $z=0$. There exist several approaches that can be used to show that many of the structures found to date indeed fulfill these requirements, as discussed below.

3.2.1 Spherical collapse model

Perhaps the simplest approach applied to protocluster data is to use the approximations of the spherical collapse model for a homogeneous sphere (Steidel et al. 1998). In this case, the total present-day mass of the protocluster is given by $M_{z=0}=\bar{\rho }V_\mathrm{true}(1+\delta _\mathrm{m})$, where $\bar{\rho }$ is the mean density of the universe, $\delta _\mathrm{m}$ is the dark matter mass overdensity, and $V_\mathrm{true}$ is the comoving volume of the protocluster in real space. The volume can be estimated from the transverse dimensions of the observed overdensity on the sky and its comoving depth corresponding to the width of the redshift distribution of the protocluster galaxies. A complication arises due to redshift space distortions, causing the overdense region to appear slightly compressed in redshift space, and the apparent overdensity to appear slightly enhanced from the true value (Kaiser 1987). The true volume is given by $V_\mathrm{app}/C$, while the mass overdensity is related to the observed galaxy overdensity, $\delta _\mathrm{gal,obs}$, through $1+b\delta _\mathrm{m}=C(1+\delta _\mathrm{gal,obs})$, where b is the bias parameter (defined as $\delta _\mathrm{gal}/\delta _\mathrm{m}$), and C accounts for the redshift space distortion of the collapsing structure. The redshift space distortion itself naturally depends on the size of the mass overdensity, $C=1+f-f(1+\delta _\mathrm{m})^{1/3}$ (Steidel et al. 1998), where $f(z)={\varOmega }_M(z)^{0.6}$ for ${\varLambda }$CDM cosmologies (Linder 2005). We are helped by the fact that the mass overdensities of protoclusters are typically still in the linear regime, with values for C in the range of 0.4–1 for $\delta _\mathrm{m}$ between 0 and 3 (see Fig. 8). For flat ${\varLambda }\hbox {CDM}$ models, C furthermore only weakly depends on redshift.

Fig. 8

Redshift space distortion (C) as a function of the matter overdensity ($\delta _\mathrm{m}$) for $z=3$. C gives the fractional compression of a protocluster volume in redshift space, and the fractional enhancement of the observed galaxy overdensity. See text for details

The spherical collapse mass overdensity can also be used to predict the future of the overdense region. By “evolving” the linear theory peak height, $\delta _\mathrm{L}$, forward in time, one can estimate at what redshift the linear overdensity will pass the critical density for collapse of a spherical symmetric perturbation ($\delta _c\simeq 1.69$; Weinberg et al. 2013). The linear overdensity $\delta _\mathrm{L}$ is directly related to $\delta _\mathrm{m}$ and can be estimated using an analytic approximation or using the result from numerical simulations (e.g., Bernardeau 1994; Mo and White 1996; Jenkins et al. 2001). $\delta _\mathrm{L}$ evolves with redshift according to $\delta _\mathrm{L}(z_2)=D(z_1)\delta _\mathrm{L}(z_1)/D(z_2)$, with $D(z)\propto g(z)/(1+z)$ and g(z) the cosmological growth factor (Carroll et al. 1992). Using this methodology, it has been shown that many of the protoclusters observed are indeed sufficiently massive and overdense that they will become genuine clusters (e.g., Steidel et al. 1998; Venemans et al. 2002; Steidel et al. 2005; Overzier et al. 2008; Cucciati et al. 2014; Lemaux et al. 2014).

3.2.2 Numerical simulations

An alternative, and perhaps more powerful, method to interpret the protocluster data is to compare the observational findings directly to the output from numerical cosmological simulations of galaxy formation (e.g., Governato et al. 1998; Suwa et al. 2006; Lucia and Blaizot 2007; Venemans et al. 2007; Overzier et al. 2009a; Angulo et al. 2012; Chiang et al. 2013, 2014, 2015; Sembolini et al. 2014; Granato et al. 2015; Muldrew et al. 2015; Contini et al. 2016; Miller et al. 2015; Hatch et al. 2016). This technique has several advantages over the analytical approximations method described above. First, state-of-the-art cosmological simulations today probe cosmological volumes of ${>}0.1$ Gpc$^3$, i.e., large enough to provide good statistical information on the properties of clusters and their high redshift protocluster progenitors (typical space densities of $10^{-9}-10^{-6}$ Mpc$^{-3}$). Second, the spherical collapse model is not an accurate description of the formation of clusters (see Fig. 1). Simulations naturally account for non-spherical collapse as well as non-linear gravitational effects. Third, in simulations (hydro, semi-analytical or abundance-matching based methods) that give predictions for the observable properties of galaxies, selection effects and other observational limitations can be taken into account. This means that we can much more accurately compare an overdensity of galaxies found at some redshift in an observational survey with similar-looking objects found in the simulations. An important caveat with these simulation-based methods, however, is that the results will depend on the accuracy of the simulation. If the cosmology or the physics of galaxies and clusters in the simulation differ significantly, one may misinterpret the data.

Chiang et al. (2013) investigated in detail the properties of the high redshift overdensities associated with ${\sim }$3000 clusters sub-divided in poor (i.e., “Fornax”-type or $1-3\times 10^{14} M_\odot$), average (i.e., “Virgo”-type or $3-10\times 10^{14} M_\odot$) and rich (i.e., “Coma”-type or $>10^{15} M_\odot$) clusters found in the Millennium Run simulation. The expected size and mass evolution of these clusters with redshift are shown in Fig. 9. The predicted comoving sizes of protoclusters (right panel) range from several to several tens of Mpc, in agreement with the observations. More massive clusters originate from larger cosmic volumes, and will thus, on average, have larger (projected) sizes on the sky. The evolution in halo mass for the most massive halo found in each (proto)cluster as a function of redshift is shown in the left panel of Fig. 9. This reveals an important result, namely that even the most massive present-day clusters first acquired a halo mass of ${\sim }10^{14} M_\odot$ only at $z\approx 2$. Wu et al. (2013) showed that such clusters have an average half-mass redshift $z_{1/2}\sim 0.6$ and experience their last major merger at $z_\mathrm{lmm}\sim 1.6$.

Fig. 9

Mass evolution for the most massive halo in a protocluster (left panel) and the size of a protocluster region (right panel) with redshift. The three colored lines indicate the median and $1\sigma$ scatter for Fornax- (blue lines), Virgo- (green lines), and Coma-type galaxy (proto-)clusters (red lines). The dotted lines indicate the results for the WMAP7 cosmology, instead of the WMAP1 cosmology. Figure reproduced from Fig. 2 in Chiang et al. (2013)

Fig. 10

Left panel Points indicate the relation between present-day cluster mass ($M_{z=0}$) and overdensity ($\delta _\mathrm{gal}$) measured for star-forming galaxies having a $\hbox {SFR} >1 M_\odot \, \hbox {year}^{-1}$ (from top-to-bottom $z=2, z=3, z=4$ and $z=5$). The median relation and $1\sigma$ scatter are indicated in red. Right panel The fraction of regions of a given overdensity that will collapse to form a galaxy cluster. The different lines indicate the predictions for overdensities of galaxies having SFRs $>1 M_\odot$ year$^{-1}$ (solid), and stellar masses of $>10^9$ (dashed) and $>10^{10} M_\odot$ (dotted). Figures reproduced from Figs. 8 and 10 in Chiang et al. (2013)

These kind of simulations are extremely powerful for interpreting the observational data of high redshift overdensities. Chiang et al. (2013) showed that there is a reasonably strong correlation between overdensities of galaxies measured at high redshift and the mass of their parent halo at the present day, at least out to $z\sim 5$ (see also Suwa et al. 2006). This is shown in the left panel of Fig. 10. Progenitors of ${\gtrsim }10^{15} M_\odot$ clusters can be identified by overdensities of star-forming galaxies of $\delta _\mathrm{gal}\gtrsim 6$ when measured on ${\sim }15$ Mpc scales. The dispersion in the $\delta _\mathrm{gal}-\mathrm {log}M_{z=0}$ relation, as suggested by the simulation, amounts to a few tenths of dex in $\mathrm {log}M_{z=0}$. The fact that $\delta _\mathrm{gal}(z)$ is a fairly accurate indicator of the final cluster mass even during this pre-collapse phase of clusters, implies that it is indeed possible to compute the present-day cluster mass based on protoclusters observed at high redshift. In agreement with the results based on the spherical collapse model, present-day masses estimated based on the much more precise calibration between the observed properties of protoclusters and $z=0$ mass in numerical simulations are $\mathrm {log}[M/M]_\odot \simeq 14-15.5$ (e.g., Governato et al. 1998; Suwa et al. 2006; Chiang et al. 2015; Lemaux et al. 2014; Cucciati et al. 2014; Hatch et al. 2016).

Muldrew et al. (2015) demonstrated that the evolutionary state of a protocluster could be determined better by computing the mass ratio of the most massive and second most massive protocluster halo. In the future, it may be possible to detect and weigh individual cores in a given protocluster based on galaxy velocity dispersions or deep X-ray observations. Alternatively, and perhaps easier to determine based on current data, one could measure the concentration of a protocluster to assess its state of collapse, analogous to the concentration parameter used in galaxy morphology studies. Two other very useful parameters that can be derived from these kind of simulations are the purity and completeness of protocluster samples. The purity is a measure of false positives of protoclusters. An example is shown in the right panel of Fig. 10. As the overdensity increases, the fraction of regions with that overdensity that corresponds to protoclusters increases as well. At $z\sim 3$, for example, it is near certain that an overdensity of $\delta _\mathrm{gal}\gtrsim 8$ will be a protocluster, while an overdensity of ${\sim }1$ will in most cases not correspond to the progenitor of a galaxy cluster (Chiang et al. 2013). Another important quantity is the completeness. This is a measure of the fraction of protoclusters that are missed by a given selection. A single object found at some z is a good (candidate) protocluster if the simulations indicate a high purity for the given selection function. A representative protocluster survey should aim to have both a high purity and a high completeness (e.g., Chiang et al. 2014; Toshikawa et al. 2016).

3.2.3 Other mass measurement methods

Besides the spherical collapse and simulation-based methods described above, mass estimates have been derived in a number of different, complementary ways for a limited but growing number of protoclusters, such as dynamical masses, stellar mass-based masses and X-ray masses. Lemaux et al. (2014) found a dynamical mass of ${\sim }3\times 10^{14} M_\odot$ at $z=3.3$. Taking the average growth rate of such a halo between $z\sim 3$ and $z=0$ then predicts a mass of ${\sim }9\times 10^{15} M_\odot$ for the $z=0$ cluster. Not only would this cluster be among the most massive clusters known today, it would also qualify as the highest redshift cluster already at $z\sim 3.3$. As discussed by Lemaux et al. (2014) it is unlikely that the system is virialized, thereby invalidating these dynamical mass estimates. However, these dynamical mass estimates provide an upper limit for the actual masses, given that the galaxies most likely populate multiple halos within the protocluster system rather than one virialized system (see also Dey et al. 2016). Venemans et al. (2007) found that the velocity dispersion of LAEs in radio galaxy protoclusters increases with decreasing redshift roughly as expected based on numerical simulations (see also Contini et al. 2016). Wang et al. (2016) detected the core of protocluster CL J1001 at $z=2.5$ in the X-rays. Assuming that the X-rays are due to a thermalized ICM, they estimate a mass of $M_{200c}=10^{13.7\pm 0.2} M_\odot$, which is corroborated by its spectroscopic velocity dispersion of $510\pm 120\, \hbox {km} \hbox {s}^{-1}$. Galametz et al. (2013) detected a (proto)cluster with a velocity dispersion of ${\sim }400-600\,\hbox {km} \hbox {s}^{-1}$ associated with the radio galaxy MRC 0156–252 at $z=2.02$ for which Overzier et al. (2005) previously detected extended soft X-ray emission with a flux ${\sim }2\times 10^{13}\, \hbox {erg} \hbox {s}^{-1} \hbox {cm}^{-2}$ of uncertain origin. However, we must be cautious of X-ray detections of (proto)clusters at these redshifts. For example, radio galaxies at high redshift are often associated with extended X-ray emission that can be confused with an ICM (Carilli et al. 2002; Overzier et al. 2005). Furthermore, there is evidence that protoclusters have an elevated AGN fraction (Sect. 4.4), which may bias the mass estimates based on low resolution X-ray data (e.g., Pierre et al. 2012). Another constraint on the cluster mass can be obtained by comparing the combined stellar mass of the galaxies in the protocluster with that of virialized clusters (e.g., Overzier et al. 2009b; Lemaux et al. 2014). Finally, as we have shown in Sect. 2.2.3, present-day cluster masses may also be estimated on the basis of Ly$\alpha$ tomographic maps that have been calibrated against accurate IGM simulations.

4 Properties of protoclusters

4.1 Protoclusters as probes of structure formation

The evolution of galaxies in protoclusters may have diverged from that of galaxies in the field already at relatively early times. Protoclusters lie at the intersection of dense, gas-rich filaments in the cosmic web (Fig. 1). It is well known that galaxies of higher stellar mass formed earlier than galaxies of lower stellar mass, and galaxies in high-density regions tend to form earlier than galaxies of the same mass in less dense regions (Thomas et al. 2005). Because the formation redshifts of cluster galaxies are high, we expect that (proto-)clusters should contain galaxies that are, on average, older and more massive compared to those in field environments at the same redshift. We may also expect differences between galaxies in and outside protoclusters, depending on the strength of environmental effects occurring preferentially in overdense regions. Galaxies in present-day clusters experienced more major mergers compared to galaxies of a similar mass outside clusters (Gottlöber et al. 2001; Fakhouri and Ma 2009). Because merging activity stimulates both AGN and star formation activity and also largely determines the morphological and kinematical properties of galaxies, some environment-driven differences may thus be expected in protocluster galaxies even at redshifts before the more classical environmental effects such as ram-pressure stripping or dynamical friction become important.

Fig. 11

The accretion history of (proto)clusters. Between $z\approx 2$ and $z\approx 3$ the core halos of protoclusters are expected to transition from a regime where the inflow of cold gas is still possible in massive halos (“cold in hot”) to a regime where all gas accreted onto massive halos shock-heats to the virial temperature (“hot”). The estimated masses of clusters and protoclusters above $z=1.5$ are indicated by large numbered symbols. The number in each symbol corresponds to the references used for the data (1: Papovich et al. (2010); Pierre et al. (2012); 2: Stanford et al. (2012); Brodwin et al. (2012); 3: Newman et al. (2014); 4: Zeimann et al. (2012); 5: Mantz et al. (2014); 6: Gobat et al. (2011, 2013); Strazzullo et al. (2013); Valentino et al. (2015); 7: Spitler et al. (2012); Yuan et al. (2014); 8: Venemans et al. (2007); Kurk et al. (2004a); 9: Steidel et al. (2005); 10: Chiang et al. (2015); 11: Diener et al. (2015); 12: Casey et al. (2015); 13: Chiang et al. (2014); Casey et al. (2015); Wang et al. (2016); 14: Venemans et al. (2007); 15: Cucciati et al. (2014); 16: Venemans et al. (2007); 17: Steidel et al. (1998); 18: Venemans et al. (2003); 19: Lemaux et al. (2014); 20: Lee et al. (2014); Dey et al. (2016); 21: Venemans et al. (2002); 22: Kuiper et al. (2011); 23: Shimasaku et al. (2003); 24: Matsuda et al. (2010); 25: Venemans et al. (2007); 26: Ouchi et al. (2005); 27: Toshikawa (2014)). Cluster data below $z=1.5$ were taken from the compilation of clusters in X-ray and SZ surveys of Bleem et al. (2015). The red dotted line traces the mass of the most massive halo of a Coma-type cluster (Chiang et al. 2013). Dashed lines trace the average halo mass growth based on the fitting functions of Behroozi et al. (2013). Thick black lines indicate the different gas cooling regimes as predicted by Dekel and Birnboim (2006)

The large-scale inflow patterns of the gas flowing into protocluster regions is also expected to be very different from that around galaxies in more average regions. The early onset and enhancement in star-formation and AGN activity in protoclusters will furthermore affect the outflow patterns of warm gas and the redistribution of metals on large scales mixing with fresh inflowing gas, while the growth of the massive cluster-sized halo will establish a region of hot gas extending out to the virial radius of the halo. The centers of protoclusters should thus be surrounded by a very complex multi-phase medium not typically encountered elsewhere. Dekel and Birnboim (2006) showed that below a critical shock-heating mass of $M_\mathrm{shock}\sim 10^{12} M_\odot$, galaxies are built by streams of cold gas (in addition to merging) that are not interrupted by virial shock-heating. For more massive halos above the shock-heating mass the situation is more complicated: at low redshift ($z\lesssim 2$) gas accreted from the IGM will immediately shock-heat to the virial temperature of the halo and will generally only be available for star formation after relatively long cooling times. At high redshift, however, cold gas is still presumed to be able to penetrate the hot gas environments in massive halos through so-called cold flows that were established prior to the formation of the shock-heated gas in these halos (see also Kereš et al. 2005, 2009; Ocvirk et al. 2008, and references therein). This general scenario has been shown to be able to reproduce many of the most essential features of galaxy evolution, such as the relatively high SFRs observed in massive galaxies at high redshift, the origin of the present-day galaxy bimodality, and downsizing (e.g., Dekel and Birnboim 2006; Cattaneo et al. 2007, 2008), although many details are still unknown and in need of observational evidence.

We propose that protoclusters are uniquely suited for testing several crucial and currently unconstrained aspects of this scenario. This is demonstrated in Fig. 11. Here we have made a first attempt to place the existing data on high redshift clusters and protoclusters from Fig. 5 on an evolutionary sequence in terms of halo mass and redshift. Accurate mass estimates exist for several of the clusters in the $1.5\lesssim z\lesssim 2$ redshift range based on X-rays, SZ effect or velocity dispersions (large red symbols in Fig. 11). In the case of the $z=2.5$ protocluster studied by Wang et al. (2016) an estimate for the mass of its massive core halo also exists based on X-rays and velocity dispersions (assuming that both trace the virialized component). For most other protoclusters it is much harder to place them on the diagram as no such mass estimates exist. However, we can estimate the mass of the most massive halos present in each protocluster by taking their estimated $z=0$ cluster mass and evolving it back in time (large blue points). To perform this extrapolation we used the mean relations between the mass of the most massive halo present in a (proto)cluster at each redshift for a given $z=0$ cluster mass based on the simulations of Chiang et al. (2013). Although this extrapolation is likely to be highly uncertain for each individual protocluster given the large uncertainties in their $z=0$ mass and relatively small number statistics, here we are mostly interested in showing the general trends based on the ensemble as a whole. We have also indicated the approximate gas cooling regimes expected for halos of a given mass and redshift [the regimes are marked “hot”, “cold” and “cold in hot” and are separated by the thick lines; from Dekel and Birnboim (2006)].

It can be seen that protoclusters occupy a very interesting region in this diagram. The shock-heating mass $M_\mathrm{shock}\sim 10^{12} M_\odot$ is believed to be relatively independent of redshift, as it relates to the halo mass above which an extended stable shock can expand to the virial radius. However, the typical mass of the halos transitioning from the cold to the hot regime have an important redshift dependence: halos of ${\sim }10^{12} M_\odot$ make the transition only at $z\sim 1$, while for halos of ${\sim }10^{13} M_\odot \,({\sim }10^{14} M_\odot$) the transition is expected to occur as early as $z\sim 2\, (z\sim 3$). The central regions of protoclusters are believed to contain the first massive halos that make the transition from the cold flows regime into the hot regime. The average (proto)cluster shown in Fig. 11 should make this transition at $z\sim 2.5$ when their core halos had masses of a few times $10^{13} M_\odot$. There are many high redshift clusters and protoclusters on either side of the diagonal line in Fig. 11 separating the two regimes. We may thus expect important changes to occur in these environments as these systems evolve from the protocluster to the cluster regime (see Sect. 6.2 for more discussion).

4.2 Evidence for a red sequence in protoclusters?

The formation of the galaxy red sequence is an extremely important and unsolved problem in galaxy evolution not exclusive to (proto)clusters. It is well established that the fraction of massive (${\gtrsim }10^{11} M_\odot$) galaxies that is quiescent is relatively constant at $0<z<1$, while the stellar mass density of quiescent galaxies of lower mass increases with decreasing redshift both in the field (Ilbert et al. 2013) and in galaxy clusters (Gilbank et al. 2008). This quenching could be related to feedback from AGN or to the build-up of hot gas reservoirs around massive galaxies in field and clusters alike. However, it has been demonstrated that there also needs to be an environmental component to the quenching process, particularly in the dense environments of galaxies in groups and clusters. The local trend of higher quiescent fractions in more denser environments or more massive halos continues toward higher redshifts, although it is considerably weaker at the highest redshifts considered ($z\sim 1-2$; Quadri et al. 2012; Scoville et al. 2013; Lee et al. 2015). Massive clusters at $z\sim 1$ generally have strong red sequences (e.g., Postman et al. 2005; Blakeslee et al. 2006; Wel et al. 2007; Patel et al. 2009; Mei et al. 2009; Muzzin et al. 2012; Foltz et al. 2015). Although the ages of the quiescent galaxies in high redshift clusters appear to be comparable to those in the field (e.g., Rettura et al. 2010; Gobat et al. 2008; Newman et al. 2014), the star formation timescales for galaxies inside clusters may be more contracted (Rettura et al. 2011). Massive red galaxies and high quiescent fractions of massive cluster members continue to be found at $1.5<z<2$ (e.g., Tanaka et al. 2010a; Papovich et al. 2010; Rudnick et al. 2012; Fassbender et al. 2014; Burg et al. 2013; Strazzullo et al. 2013; Andreon et al. 2014; Newman et al. 2014; Koyama et al. 2014; Cooke et al. 2016; Hatch et al. 2017), although these systems also frequently host star-forming or starbursting galaxies not seen at the lower redshifts (e.g., Hilton et al. 2010; Tran et al. 2010; Fassbender et al. 2011; Gobat et al. 2013; Brodwin et al. 2013; Strazzullo et al. 2013; Lee et al. 2015; Mei et al. 2015; Wang et al. 2016) (but see Cooke et al. 2016; Andreon et al. 2014; Newman et al. 2014).

The evolution of the cluster red sequence is the net result of several competing or complementary processes responsible for either driving or shutting down star formation in galaxies. The dominant mechanisms are expected to differ across redshift, stellar mass, halo mass and environment, and it is still a challenge for models to reproduce the quiescent cluster galaxy population observed at high redshift (e.g., Sommariva et al. 2014). Galaxy (proto-)clusters offer a unique chance to investigate the red sequence formation because they allow us to study dense regions out to higher redshifts, i.e., closer to the epoch of formation and quenching of the massive cluster galaxies. What is the evidence for red sequence galaxies in the sample of protoclusters available to date? One of the first objects that allowed us to address this question is the protocluster associated with the Spiderweb galaxy, a radio galaxy at $z=2.2$. Kurk et al. (2004b) identified an excess of photometrically selected extremely red objects (EROs) coinciding with the protocluster. They concluded that most of the EROs were likely to be dust-enshrouded starburst galaxies but marked 5 galaxies as potential quiescent red sequence objects. Zirm et al. (2008) performed the first deep near-IR study of this field, using HST/NICMOS to search for objects having relatively red $J-H$ colors (approximately rest-frame $U-B$) indicative of a forming red sequence. Although still working with photometrically selected (i.e., not spectroscopically confirmed) samples, they found a factor of 6 overdensity of relatively red objects over a small area surrounding the protocluster compared to similar observations performed in the HUDF. They showed that these galaxies were bluer than galaxies on the red sequence in both the Coma cluster and in RDCS 1252.9–2927, a cluster at $z=1.24$ used for reference. However, the objects were redder than expected based on passively de-evolving the RDCS 1252.9–2927 red sequence to $z=2.2$ [this assumed a formation redshift of $z\sim 3$ based on the analysis of Blakeslee et al. (2003)]. Although this could indicate that the Spiderweb protocluster contains a passive population older than that expected for an RDCS 1252.9–2927-like cluster, it cannot be ruled out at the moment that the red excess is due to dusty star-forming galaxies. Zirm et al. (2008) also showed that their brightest and reddest objects have high Sersic indices ($n=2.7-5$) indicative of early-type morphologies, which supports the conclusion that at least some of these objects are genuine quiescent galaxies rather than dusty galaxies. Kodama et al. (2007) explored the red populations toward 4 regions at $2\lesssim z\lesssim 3$, previously identified as protoclusters based on excesses of LAEs and HAEs near radio galaxies. They found that the bright, red end of the CMRs are more populated with respect to comparison field data. The excesses are particularly strong for the targets at the low redshift end ($z\lesssim 2.5$) of their small sample, indicating that a red sequence of massive, passively evolving galaxies was first established between $z\sim 2$ and $z\sim 3$ (with $z_\mathrm{f}\simeq$ 3.5–5). Similar results based on near-IR color cuts in the fields of other radio galaxies at these redshifts were obtained also by other authors (e.g., Kajisawa et al. 2006; Hatch et al. 2011a).

Although the excesses of red galaxies observed in these structures are promising, we must be cautious not to overinterpret these results as they are based on photometric selections only. Doherty (2010) obtained a large number of optical and near-IR spectra of red galaxies in two protoclusters that had previously been shown to have rather prominent excesses of red galaxies, including the Spiderweb (Kurk et al. 2004b; Kodama et al. 2007; Zirm et al. 2008). Out of 90 objects targeted, membership of the $z=2.2$ cluster could be confirmed for just two red objects. Of these, one object was found to be a star-forming galaxy reddened by dust (based primarily on a 24 $\upmu$m detection), while the other could be a genuine quiescent galaxy (based on a low SFR based on H$\alpha$ and a high stellar mass based on SED fitting). Both objects were found to have stellar masses of a few times $10^{11} M_\odot$, and are thus suitable candidates for becoming massive red sequence cluster galaxies (unless they merge with the BCG, see Doherty 2010). By combining photometric data with low S/N spectroscopic data (e.g., Kriek et al. 2006), Tanaka et al. (2013) were able to further constrain the redshifts and physical properties of the red population. Assuming protocluster membership, they showed that there exists a small population of massive, quiescent galaxies ($\hbox {sSFR}\,{<}10^{-11}\,\hbox {year}^{-1}$) consistent with a formation redshift of $z_\mathrm{f}\simeq \,$3–4 and a relatively short star formation timescale ($\tau \lesssim 0.5\,\hbox {Gyr}$).

It may be clear from the above that the study of red sequence formation in protoclusters is severely limited by the lack of spectroscopic redshifts available for the candidate red cluster members. In part this is due to protocluster surveys and selection techniques relying often on star-forming galaxy populations rather than on quiescent galaxies to identify the protoclusters, and in part due to the difficulty in obtaining large numbers of spectroscopic redshifts of relatively quiescent galaxies at high redshift. One exception is the study of Lemaux et al. (2014) that found a highly overdense region ($\delta _\mathrm{gal}\approx 11$) in a large spectroscopic survey (see Sect. 3.2). VUDS is a relatively wide-field (${\approx }3$ deg$^2$) survey that targets galaxies having a high likelihood of being at high redshift ($2\lesssim z\lesssim 6$) based on their photometric redshifts, rather than using a magnitude cut. At the redshift of the protocluster Cl J0227–0421, the VUDS survey is sensitive to galaxies down to a stellar mass limit of ${\sim }10^9 M_\odot$, allowing an investigation of the colors of spectroscopically confirmed protocluster and field galaxies as a function of stellar mass. The most notable finding is a significant excess of relatively red galaxies having high stellar masses of ${\sim }10^{11} M_\odot$ in the protocluster compared to the field. Although the absolute number of massive, red galaxies is small (three, excluding the proto-BCG which is equally massive but has blue colors), they represent a factor ${\sim }25$ overdensity compared to the field. The excess is still a factor of ${\sim }3$ when taking into account that the protocluster itself is overdense compared to the field. Lemaux et al. (2014) refer to this population as “proto-red sequence galaxies” as they have masses consistent with those of red sequence galaxies in low redshift clusters, and colors and stellar populations that indicate no recent major star formation since ${\sim }300\,\hbox {Myr}$, even though they are not strictly “passive”. This implies formation redshifts of these stellar populations of $z\sim 4$, i.e., similar to that derived for massive cluster galaxies at low redshift. The excess of massive galaxies as found in Cl J0227–0421 confirms the trend seen in many protoclusters in the redshift range $z\sim 2-3$, but the additional excess of relatively red, possible (proto-)red sequence galaxies, has so far been unambiguously demonstrated in only very few systems (e.g., Lemaux et al. 2014; Wang et al. 2016) (but see Diener et al. 2013; Cucciati et al. 2014, for counter evidence also based on a large number of spectroscopic redshifts in the zCOSMOS and VUDS surveys).

Contini et al. (2016) investigated the growth of the red sequence in protoclusters having present-day masses of ${\sim }10^{15} M_\odot$ in the context of semi-analytic models. These authors predict that at $z\sim 3, \sim$90% of the progenitors of cluster red sequence galaxies are still actively star-forming. The passive red sequence forms primarily through the quenching of satellite galaxies as they deplete their cold gas reservoir after being accreted onto the cluster. Satellite galaxies accreted at higher redshifts ($z\simeq 2-3$) have higher SFRs and quench faster compared to satellites that are accreted at lower redshifts ($z\simeq 1-2$). This is because galaxies at higher redshifts have higher gas fractions and lower mass halos where outflows are more effective at removing cold gas. For central galaxies, the quenching occurs much later ($z<1$, with ${\sim }$20% quenched by $z=0.5$), and is primarily caused by AGN feedback that offsets the cooling of the hot gas. The models of Contini et al. (2016) predict that the quiescent fraction in (proto)clusters is no more than 20% at $z\sim 3$, about 30% at $z\sim 2$ and about 60% at $z\sim 1$. These numbers appear to be consistent with the data for protoclusters as well as most $z<2$ clusters, except perhaps for JKCS 041 at $z=1.80$ which has an extraordinary quiescent fraction of 80% (Newman et al. 2014). This may indicate that cosmic variance or assembly bias are important factors when studying (proto)clusters at these redshifts, highlighting the importance of increasing the sample size. Newman et al. (2014) found no differences in the ages of the stellar populations of the red sequence galaxies compared to a sample of co-eval galaxies in the field. This suggests that the epoch of quenching of star-formation in massive galaxies is similar for clusters and the field, although the process may be more efficient in clusters given the large number of quenched galaxies found, at least in JKCS 041 (Newman et al. 2014).

4.3 Brightest cluster galaxies

Brightest cluster galaxies are the most massive galaxies at the centers of galaxy clusters. As shown in Lin and Mohr (2004), they are typically located close to the cluster center as defined by the peak X-ray surface brightness. They are more massive in more massive clusters, and contain a smaller fraction of the total light with increasing cluster mass. BCGs accumulate their mass primarily through mergers, and the mass fraction of stars contained in the ICM, believed to be a by-product of such mergers, increases with increasing cluster mass (Lin and Mohr 2004). The largest type of BCGs, the ‘cD’ galaxies, often contain secondary or multiple nuclei indicative of their cannibalistic past (see Tonry 1987). BCGs have several important structural and chemical properties distinct from non-BCGs of the same stellar mass, and more frequently host radio-loud AGN (Linden et al. 2007; Best et al. 2007). Their formation history is not only governed by the evolution of massive galaxies in general, but likely also depends on the complexities of cluster formation and significant environmental effects. By comparing the evolution of BCGs as observed in matched cluster samples at different epochs with the evolution predicted by simulations, we can test both the large-scale cluster assembly process and the detailed assembly process of the BCG in the cluster centers (e.g., Lucia and Blaizot 2007; Collins et al. 2009; Lin et al. 2013; Inagaki et al. 2015; Zhao et al. 2015).

Protoclusters offer a unique opportunity to observe the initial stages of the development of BCGs, which can provide some much needed constraints for models and numerical simulations that are relatively untested at these high redshifts. It is expected that BCG formation will be accompanied by excessive merging, star formation and AGN activity in the dense protocluster environment, especially at high redshift. The general class of high redshift radio galaxies have many properties indicative of progenitors of local BCGs (e.g., Pentericci et al. 1997; Best et al. 1998; Zirm et al. 2005; Miley and Breuck 2008; Hatch et al. 2009; Overzier et al. 2009b; Collet et al. 2015). Perhaps one of the most spectacular examples of such a proto-typical BCG found to date is the ‘Spiderweb galaxy’, a radio galaxy at the center of a $z=2.2$ protocluster. This system has many intriguing features that both quantitatively and qualitatively suggest that it is a forming BCG (see Figs. 12, 16). Its main component in the rest-frame optical has a classical De Vaucouleurs profile, while its larger-scale morphology in the rest UV and optical is extremely clumpy and contains numerous faint satellites and diffuse emission (Pentericci et al. 1997, 1998; Miley et al. 2006). It has an exceptionally large stellar mass (${\sim }10^{12} M_\odot$; Pentericci et al. 1997; Hatch et al. 2009), a high molecular mass ($M_{H_2}\sim 10^{11} M_\odot$; Emonts et al. 2013), a high SFR (${>}1000 M_\odot \,\hbox {year}^{-1}$; Stevens et al. 2003; Hatch et al. 2008; Seymour et al. 2012), and an SMBH of ${\sim }10^{10} M_\odot$ (Nesvadba et al. 2011). The radio AGN shows evidence for strong interaction with the surrounding medium (Pentericci et al. 1997), capable of removing a large fraction of the gas and quenching star formation (Nesvadba et al. 2006). On the scale of present-day cD galaxy envelopes, the system shows a luminous emission line nebula (Pentericci et al. 1997; Kurk et al. 2000), extended star formation resembling a young version of the ICL (Pentericci et al. 1998; Hatch et al. 2008), a dense (magnetized) gaseous medium with high Faraday rotation measure (Pentericci et al. 1997), and extended soft X-ray emission due to shocked gas or a faint thermal ICM (Carilli et al. 2002). Saro et al. (2009) performed a set of hydrodynamical simulations that followed the growth of the BCG in poor and rich clusters (Fig. 12). By comparing the SFR, mass and morphology of the BCGs and the luminosity functions and velocity dispersions of the protoclusters, Saro et al. (2009) found that the properties of the Spiderweb protocluster span the range of properties between the two simulations. They also showed that these systems are already surrounded by a significant fraction of ICL resulting from the early merging activity of the proto-BCG.

Fig. 12

Observations and simulations of proto-BCGs. Left panel: rest-frame UV image of the Spiderweb Galaxy at the center of a $z\approx 2.2$ protocluster (Miley et al. 2006). Right panel: simulated image of the stellar mass density around a proto-BCG in a forming galaxy cluster at $z=2.1$ (Saro et al. 2009). The excesses of small and large satellites, the centrally dominant massive galaxy and the diffuse extended nebula of stars are in qualitative agreement with the properties of the Spiderweb Galaxy shown left. The images are shown at the same scale, and measure 150 $h^{-1}$ physical kpc in diameter. The image shown in the right panel was taken from Fig. 1 in Saro et al. (2009)

Although the Spiderweb galaxy appears to be relatively exceptional compared to other systems known at the present moment, several other protoclusters show evidence for BCGs in various stages of formation, such as massive, quiescent galaxies at the centers of dense groups in protoclusters (e.g., Spitler et al. 2012; Yuan et al. 2014; Belli et al. 2014; Hatch et al. 2017), tight groups of central interacting galaxies (Gobat et al. 2011, 2013; Strazzullo et al. 2013), massive AGN host galaxies (Lemaux et al. 2014; Hennawi et al. 2015), and dense groups of massive quiescent or (dusty) star-forming galaxies having dynamical or total stellar masses consistent with those expected for present-day BCG progenitors (e.g., Kubo et al. 2016; Ishigaki et al. 2016). However, it is also important to note that there are many examples of protoclusters in which no particular galaxy members are present, or in which they have not yet been identified (e.g., Cucciati et al. 2014; Steidel et al. 2005; Ouchi et al. 2005; Toshikawa 2014; Dey et al. 2016).

It is tempting to draw conclusions regarding the early evolution of BCGs by connecting, e.g., the Bootes protocluster at $z=3.78$ that contains no particular members (Lee et al. 2014; Dey et al. 2016), the compact group of massive and starbursting galaxies in the core of the SSA 22 protocluster at $z=3.1$ (Kubo et al. 2016), the centrally dominant and highly active ‘Spiderweb Galaxy’ at $z=2.2$, and the relatively quiescent systems seen in clusters at $z\lesssim 2$. However, given the small sample sizes and the hugely varying selection techniques used, we must be very careful not to overinterpret the data at this early stage. It can be instructive to compare the observational evidence with simulations of BCG formation. Lucia and Blaizot (2007) used a semi-analytic model to specifically study BCG formation (see also Dubinski 1998; Gao et al. 2004). One of the key predictions of that study is a large discrepancy between the assembly history (defined as the mass growth of the main progenitor of the BCG with time) and the star formation history (SFH) of the BCG (defined as the sum of the SFHs of all the progenitor galaxies that merge to form the BCG). More specifically, they found that, on average, 50 and 90% of the stars in present-day BCGs formed at $z\gtrsim 4$ and $z\gtrsim 2$, respectively. In comparison, the redshifts at which 50 and 90% of the stellar mass in these BCGs were assembled into a single galaxy are much lower, at $z\sim 0.5$ and $z\lesssim 0.2$, respectively. Prior to $z\sim 1$, the main BCG progenitor is typically difficult to identify, especially since it is not necessarily the most massive galaxy in the protocluster. This picture is, at least qualitatively, consistent with those protoclusters in which no obvious proto-BCGs have been identified, or those in which there appear to be several massive (central) objects. Although the relatively late assembly of BCGs as suggested by these simulations appears to be in disagreement with protoclusters that already show evidence for a significant central object (e.g., Pentericci et al. 1997; Overzier et al. 2009b; Lemaux et al. 2014; Hennawi et al. 2015), it is worth highlighting another important feature of the models: when studying the properties of the central galaxies in the most massive haloes selected at high redshift rather than at the present-day, Lucia and Blaizot (2007) and Angulo et al. (2012) found that these are generally not the progenitors of the most massive haloes at $z=0$ (although they are still found in clusters). It is not unlikely that the biased tracer technique used to find protoclusters near radio galaxies and quasars preferentially selects systems in which the central galaxies are in a more advanced stage compared to those in the more typical progenitors of clusters of the same mass. This could be tested with larger samples of protoclusters and a better understanding of the selection effects.

These massive (forming) galaxies in protoclusters offer new insights into the progenitors of the BCGs in local and high redshift clusters. Common characteristics include high stellar masses, high SFRs, significant AGN activity and feedback, complex systems in the process of merging, and extended reservoirs of ionized or molecular gas. As the sample of high redshift clusters and protoclusters expands in the coming decade, we will be able to track the growth of the BCGs in these systems and provide valuable input for models of the formation of this important class of galaxies.

4.4 Other aspects of protocluster galaxies

The protocluster data available today, although still limited to a few well-studied systems, are starting to offer us a unique chance to study galaxy formation in a regime that was, until recently, only accessible in theory and simulations. Here we will briefly review some observational results related to the morphologies, stellar populations, gas-phase abundances and AGN fraction of galaxies in protoclusters.

4.4.1 Morphologies of protocluster galaxies

Protoclusters not only allow us to study massive galaxy formation in general (protoclusters contain large numbers of co-eval galaxies), they have also allowed some of the first quantitative studies of galaxy morphology in dense regions at $z\gtrsim 2$. Some protoclusters were extensively imaged with HST, providing size and morphology measurements in the rest-UV for large numbers of LAEs and LBGs (e.g., Miley et al. 2004; Venemans et al. 2005; Overzier et al. 2006, 2008; Peter et al. 2007). These studies showed the typical range of structures known for these kind of objects consisting of single, double, or multiple knotty structures with relatively small effective radii of, on average, ${\sim }1$ kpc. The sizes of the LAEs are generally smaller than those of the LBGs, which reflects the fact that they are also fainter and younger galaxies, but this is no different in protoclusters and in the field (Venemans et al. 2005; Peter et al. 2007; Overzier et al. 2008). Miley et al. (2006) observed a higher frequency of chain and tadpole morphology star-forming galaxies close to the Spiderweb Galaxy, the presumed proto-BCG of a protocluster at $z=2.2$, compared to the field. Hine et al. (2016) found an enhanced major merger rate in the $z=3.1$ SSA 22 protocluster based on visual classification and pair counts in the rest-UV. Such an enhanced merger rate could lie at the basis of a number of other environmental effects seen in this region, such as an enhanced AGN fraction (Lehmer et al. 2009) and an excess of both star-forming and quiescent massive galaxies (Kubo et al. 2013, 2015; Casey 2016).

Some studies have also reported that early-type galaxies in the highest redshift (proto-)clusters appear larger than those in the field. In principle, some degree of accelerated structural evolution may be expected given the much higher pair counts and inferred merger rates in the overdense environments of (proto-)cluster galaxies (Lotz et al. 2013). Consistent with such a higher merger rate, Zirm et al. (2012) find an indication that massive, quiescent galaxies in a $z=2.2$ protocluster are larger, and thus less dense, and have a higher Sérsic parameter than field galaxies of similar mass. Similar findings have been made for other $z\sim 2$ (proto-)clusters (Papovich et al. 2012; Strazzullo et al. 2013), while no differences in the structural properties of massive quiescent galaxies were found in a massive cluster at $z=1.80$ (Newman et al. 2014). Newman et al. (2014) conclude that the combined evidence from all (proto-)cluster data combined is rather weak, but point out that difficulties arising from systematic differences in measurement techniques and working with non-spectroscopic samples do not rule out that some real structural differences already exist at this early epoch. Simulations of protocluster regions show that such environments may also be expected to contain massive galaxies that are more compact (denser) compared to those in the field, and that such galaxies require significant “puffing up” in order to comply with local structural relations for massive galaxies (Sommer-Larsen and Toft 2010). Wang et al. (2016) found that both quiescent and star-forming massive members in the core of protocluster CL J1001 at $z=2.5$ are more compact compared to local cluster early-types, but no different compared to field galaxies at the same redshift. Thus, this may indicate that any of the changes that may occur to the sizes of galaxies in dense environments have not yet occurred.

Fig. 13

Stellar ages (left panel) and stellar masses (right panel) of the UV-selected galaxies in and outside the protocluster HS1700+643 at $z=2.30$ discovered by Steidel et al. (2005). The protocluster contains galaxies that are both more massive and older than galaxies at neighboring redshifts. Data taken from Fig. 3 in Steidel et al. (2005)

Fig. 14

The relation between stellar mass and H$\alpha$-based SFR for galaxies in (proto-)clusters and the field at three different redshifts: $z=0.4$ (left panel), $z=0.8$ (middle panel) and $z=2.2$ (right panel). Although significant evolution in the so-called main sequence of star formation with respect to the $z=0$ relation (dot-dashed lines) is seen for all three redshifts, the relation for field and (proto-)cluster regions is indistinguishable, except for a possible enhancement in the number of galaxies at the high mass end in (proto-)clusters. Figure reproduced from Fig. 8 in Koyama et al. (2013b)

4.4.2 Stellar populations and star formation

We have already discussed the available evidence on red sequence galaxies in protoclusters in Sect. 4.2. Most notably perhaps were the excesses of massive, quiescent galaxies based on spectroscopic data (e.g., Tanaka et al. 2013; Lemaux et al. 2014). Steidel et al. (2005) found that the star-forming galaxies in the protocluster HS1700+643 at $z=2.30$ are both more massive and older than an unbiased sample of field galaxies selected from neighboring redshifts (Fig. 13). Although this protocluster is not known to contain a cluster red sequence, these results are among the strongest indications that the properties of galaxies in some protoclusters differ strongly from those in the field. Based on a large sample of HAEs in the Spiderweb protocluster at $z=2.2$ (Koyama et al. 2013a) and a control sample of HAEs in a large field survey, Koyama et al. (2013b) showed that the SFR-$M_*$ relation is indistinguishable between protocluster and field environments (Fig. 14; see also Cooke et al. 2016). However, a small excess of massive ($M_*>10^{11} M_\odot$) galaxies relative to the field is seen, and these galaxies tend to also have higher SFRs relative to the field (see also Kurk et al. 2009; Hatch et al. 2011b). Hatch et al. (2011b) also find that $z=2.2-2.4$ protoclusters (including the Spiderweb protocluster) contain HAEs that are, on average, twice as massive as HAEs in the field. They do not find any difference in their SFRs, but it is possible that the subtle effect reported by Koyama et al. (2013b) could only be seen due to their much larger HAE sample size. Cooke et al. (2014) furthermore found an excess of massive galaxies at $z=2.5$, while at the same time showing a rather puzzling deficit in the relative number of low-mass star-forming galaxies.

The results summarized above shed some light on galaxy evolution during the epoch when the first differences between field and clusters started to appear. For completeness, however, we should also point out the rather large subset of protoclusters for which no differences are seen compared to the field, although these tend to be the higher redshift systems at $z\gtrsim 4$ where the environmental effects are expected to be even smaller than at $z=2-3$ and more difficult to measure (e.g., Overzier et al. 2008; Toshikawa 2014; Toshikawa et al. 2016; Lee et al. 2014). In the cases where significant differences were found between protocluster and field galaxies, these differences are most naturally explained by taking into account that protocluster regions had an earlier start compared to the field. For example, Steidel et al. (2005) interpret the large age and mass difference as a consequence of assembly bias, given that the age difference between protocluster and field galaxies is close to the age difference one would naively expect on the basis of the overdensity and the earlier formation time of the galaxies forming within the overdensity. Results obtained in other protoclusters also argue in favor of an accelerated growth in overdense regions compared to the field (e.g., Hatch et al. 2011b; Matsuda et al. 2011; Koyama et al. 2013b; Cooke et al. 2014). However, we must remain somewhat cautious to compare results obtained with different field sizes or non-spectroscopic samples. Also, the possible contamination by AGN or improper dust corrections could easily skew the results towards or away from the true relations, as the sample of available protoclusters is still rather small.

Despite the rather delicate evidence for environmental trends in terms of stellar populations in protoclusters discussed above, there is strong evidence that protoclusters are often associated with large excesses of dust-obscured star formation in the $z\simeq 2-4$ redshift range (e.g., Stevens et al. 2003; Breuck et al. 2004; Greve et al. 2007; Priddey et al. 2008; Daddi et al. 2009; Carrera et al. 2011; Ivison et al. 2013; Casey et al. 2015; Planck Collaboration et al. 2015; Valtchanov et al. 2013; Dannerbauer et al. 2014; Rigby et al. 2014; Umehata et al. 2014, 2015) (see Casey 2016, for an overview). The high levels of dust-obscured star formation in protoclusters suggests that they are rapidly forming massive galaxies. Protoclusters rich in dusty galaxies perhaps represent a particularly important phase during which a large fraction of the stellar mass on the present-day cluster red sequence was being formed during only a very brief time span in accordance with the small scatter of the cluster red sequence (e.g., Mei et al. 2009). The tail of this starburst phase can still be seen in some high redshift clusters which show unusual starbursts typically not seen in lower redshift clusters (e.g., Webb et al. 2013, 2015a, b; Lee et al. 2015; Wang et al. 2016). As the starburst activity in (proto)clusters declines with decreasing redshift, the fraction of normal quiescent galaxies is seen to rise quite rapidly from $z\sim 2.5$ to $z\sim 1.5$ (e.g., Newman et al. 2014; Cooke et al. 2015; Wang et al. 2016), indicating that the main epoch of massive cluster galaxy formation has ended.

Fig. 15

Compilation of stellar mass versus gas-phase oxygen abundance measurements of star-forming galaxies made in different environments in high redshift (proto)clusters. Measurements made at different stellar masses in the same systems are connected by dotted lines. The error bars on the data include both the statistical errors on the protocluster measurements and those on the field measurements. See the discussion in the text for details

4.4.3 Gas-phase metal abundances

Recent advances in instrumentation for performing deep multi-object spectroscopy in the near-infrared has allowed significant progress in the study of the nebular emission line properties of star-forming galaxies at high redshift. In Fig. 15 we show a compilation of some of the first results related to the gas-phase oxygen abundances of galaxies in several high redshift (proto)clusters. Relatively low-mass ($M_*\sim 10^{10} M_\odot$) galaxies in three protocluster regions at $z\approx 2.2, z\approx 2.3$ and $z\approx 2.5$ have marginally higher mean gas-phase metal abundances compared to field galaxies at the same redshift, while no such difference is seen for high mass ($M_*\sim 10^{11} M_\odot$) galaxies (Kulas et al. 2013; Shimakawa et al. 2015). In contrast, other studies have either found no differences between galaxies in (proto-)clusters and in the field (Kewley et al. 2016; Kacprzak et al. 2015; Tran et al. 2015) or lower abundances in protoclusters (Valentino et al. 2015). Of course, it is still difficult to synthesize a consistent story based on these preliminary results on just a handful of systems. However, detailed investigation of the abundances of galaxies as a function of stellar mass and redshift in dense (proto)cluster environments and the field will be of great value to galaxy formation models. For example, enhancements in the metallicities of galaxies in protoclusters may be expected if these galaxies are accreting their gas from an IGM that is already metal-enhanced compared to lower density regions. The mass-dependent metallicity offset detected by Kulas et al. (2013) and Shimakawa et al. (2015) could then be the result of the more efficient recycling of enriched gas for low-mass galaxies in overdense regions which enhances the recycling rate of outflowing metal-enriched gas, as suggested by numerical simulations (Davé et al. 2011; Oppenheimer and Davé 2008). The mean metallicity could also be increased due to stripping of the outer, more metal-poor gas through merging or ram-pressure stripping which should be more efficient in dense regions (Shimakawa et al. 2015; Kacprzak et al. 2015), while a dilution of the metallicity could also be expected due to the efficient inflow of gas from the IGM in the form of cold streams or mergers (Giavalisco et al. 2011; Cucciati et al. 2014; Valentino et al. 2015).

4.4.4 AGN

Several studies have found evidence for an elevated AGN fraction among protocluster galaxies (e.g., Pentericci et al. 2002; Lehmer et al. 2009; Digby-North et al. 2010; Lehmer et al. 2013; Chiang et al. 2015). Such increased AGN activity could be caused by the enhanced inflow rates of gas onto galaxies in overdense regions, either from the IGM or through galaxy merging. In addition, an elevated fraction of AGN is expected if protoclusters contain more massive galaxies relative to the field, given that the AGN fraction rises strongly as a function of stellar mass. The increased AGN fraction in protoclusters could be a reversal of the trend seen in groups and clusters at lower redshift, that tend to have AGN fractions that are either lower or equal to that in the field at $z\sim 0$ and $z\sim 1$, respectively (Martini et al. 2013).

5 Connections with Ly$\alpha$ blobs, radio galaxies, QSOs, and reionization

5.1 Protoclusters and Ly$\alpha$ blobs

The enigmatic class of giant emission line nebulae commonly referred to as Ly$\alpha$ halos (LAHs; when seen around radio galaxies and QSOs) and Ly$\alpha$ blobs (LABs; in other cases) has been closely intertwined with the subject of protoclusters since their first discovery (e.g., Francis et al. 1996; Fevre et al. 1996; Steidel et al. 2000; Venemans et al. 2002; Matsuda et al. 2004). These nebulae emit extremely luminous ($L_{\mathrm{Ly}\alpha }\sim 10^{43-45}$ erg s$^{-1}$) Ly$\alpha$ emission that is extended over tens or sometimes hundreds of kpc (see Fig. 16). They frequently occur around radio galaxies (e.g., Heckman et al. 1982; Baum and Heckman 1989; McCarthy 1993; Ojik et al. 1996; Breuck et al. 2000; Overzier et al. 2001; Venemans et al. 2002; Villar-Martín et al. 2007a), radio-loud QSOs (e.g., Heckman et al. 1991; Borisova et al. 2016), radio-quiet QSOs and obscured AGN (e.g., Basu-Zych and Scharf 2004; Chapman et al. 2004; Christensen et al. 2006; Dey et al. 2005; Geach et al. 2009; Yang et al. 2009; Fu and Stockton 2009; Overzier et al. 2013; Bridge et al. 2013; Cantalupo et al. 2014; Hennawi et al. 2015; Borisova et al. 2016), star-forming galaxies (e.g., Ouchi et al. 2009; Colbert et al. 2011), and sometimes without a clearly identifiable source (e.g., Steidel et al. 2000; Prescott et al. 2008; Erb et al. 2011). Extended Ly$\alpha$ emission in galaxies having more typical luminosities of $L_{Ly\alpha }\sim 10^{42-43}$ erg s$^{-1}$ (also referred to as LABs by some authors) is most likely due to scattered Ly$\alpha$ photons from central or extended star formation, AGN, and perhaps some amount of gravitational cooling radiation (e.g., Hayashino et al. 2004; Saito et al. 2006; Zheng et al. 2011; Steidel et al. 2011; Dijkstra and Kramer 2012; Matsuda et al. 2012; Momose et al. 2014, 2016; Caminha et al. 2015; Patrício et al. 2016). In the case of the nebulae around radio galaxies and QSOs, there is little doubt that the powerful AGN are the main driver of the Ly$\alpha$ phenomenon. This appears to be true for the majority of all other LABs as well (at least at high luminosities, $L_{\mathrm{Ly}\alpha }\gtrsim 10^{44}$ erg s$^{-1}$), given that most LABs host a luminous obscured AGN (see the overview in Overzier et al. 2013). Although luminous high redshift AGN are often surrounded by powerful starbursts that can provide additional ionizing photons (e.g., Stevens et al. 2003; Villar-Martín et al. 2007a; Hatch et al. 2008), even in these sources the dominant power source of their Ly$\alpha$ emission is the photoionization by the central AGN. Cases of LABs that are purely powered by star formation or gravitational cooling radiation must be very rare, otherwise many non-AGN LABs would be detected, especially considering that star formation and cooling radiation have significantly longer duty-cycles compared to AGN.

Fig. 16

Extended Ly$\alpha$ emission nebulae. Panels show narrow-band Ly$\alpha$ images of the radio-quiet QSO UM287 at $z\approx 2.3$ [left panel figure reproduced from Fig. 1a in Cantalupo et al. (2014)] and the Spiderweb radio galaxy at $z=2.2$ [right panel Kurk et al. (2000)]. Both images have a diameter of $2^\prime$, and the AGN are located at the center of each panel

It is intriguing that many protoclusters and overdense regions at high redshift known to date also host halos or LABs. In the case of protoclusters found near powerful high redshift radio galaxies this is obvious, given that these large extended emission line regions are conspicuous among these galaxies (McCarthy 1993). However, LABs have been found also in other overdense regions not associated with radio galaxies (e.g., Francis 2001; Steidel et al. 2000; Matsuda et al. 2004, 2009; Saito et al. 2006; Prescott et al. 2008, 2012; Erb et al. 2011; Hayashi et al. 2012; Cantalupo et al. 2014; Hennawi et al. 2015; Chiang et al. 2015), suggesting that the connection between protoclusters and large Ly$\alpha$ nebulae could be more fundamental. The fact that the largest and most luminous halos and LABs are nearly always associated with AGN (radio-loud and radio-quiet obscured AGN and QSOs) points to a relation between protoclusters and (powerful) AGN. This connection can be understood, at least at the qualitative level. AGN are powered by the SMBHs in massive galaxies. The formation of these SMBHs relies on the merging between galaxies, while the triggering of the AGN requires a large supply of cold gas in their nuclear region. This gas could be supplied by mergers, or through infall from the circumgalactic medium. Overdense environments at high redshift are inducive to frequent mergers between (massive) galaxies, and are expected to be associated with high densities of cold gas recently accreted from the IGM.

Besides the triggering of the AGN, we also need to explain the presence of the gas on scales of tens or hundreds of kpc around the AGN in the cores of protoclusters. Ly$\alpha$ halos and LABs frequently show highly perturbed or outflowing gas in their inner parts (and along radio jets, if present), while infall or rotating gas dominates the outer parts (e.g., Ojik et al. 1996; Nesvadba et al. 2006; Villar-Martín et al. 2007b; Humphrey et al. 2008; Martin et al. 2014, 2015; Prescott et al. 2015). The extended gas reservoirs around radio galaxies such as the Spiderweb Galaxy could perhaps in large part be explained by strong mechanical feedback from the radio jets (Nesvadba et al. 2006). However, the fact that perturbed gas occurs both in radio-loud and radio-quiet sources indicates that feedback does not necessarily require radio jets, but can also arise from QSO or starburst winds (Polletta et al. 2011; Liu et al. 2013; Overzier et al. 2013; Hennawi et al. 2015). In this case, it is unclear whether the presence of gas found at large distances from QSOs (and some radio galaxies) is consistent with the outflow scenario (e.g., Prochaska and Hennawi 2009; Cantalupo et al. 2014; Johnson et al. 2015; Hennawi et al. 2015). Quasar hosts are massive galaxies that live in massive dark matter halos (${\sim }10^{12-13} M_\odot$) that should already be filled with substantial amounts of shock-heated gas of temperatures of ${\sim }10^7$ K at $z\sim 2$ (e.g., Birnboim and Dekel 2003; Dekel and Birnboim 2006; Ocvirk et al. 2008; Cattaneo et al. 2008; Fumagalli et al. 2014, see also Fig. 11). A possible scenario is thus that the Ly$\alpha$ emitting clouds form within the hot plasma in the QSO halos (Mo and Miralda-Escude 1996). In some systems observed, the QSOs appear to illuminate gas in the IGM, possibly the accretion flows, given that the physical extent of some nebulae exceed the virial radius of their dark matter halos (Cantalupo et al. 2014; Martin et al. 2015; Borisova et al. 2016). Another possibility is that the Ly$\alpha$ emission arises from gas associated with the debris of massive mergers that is being illuminated by the AGN (Yajima et al. 2013; Yang et al. 2014; Hennawi et al. 2015; Johnson et al. 2015). Ly$\alpha$ halos and LABs often show substantial chemical enrichment (Breuck et al. 2000; Overzier et al. 2001; Prochaska and Hennawi 2009; Hennawi et al. 2015), which would perhaps be most easily explained by the outflow or merger scenario, rather than a scenario where the gas originates from the IGM.

In addition to the luminous Ly$\alpha$ systems discussed here, Matsuda et al. (2012) showed that also the relatively small and faint Ly$\alpha$ emission halos seen around ordinary star forming galaxies may contain important information about the detailed structure of protoclusters. They suggest that in overdense regions the strong clustering between dark matter halos may lead to a different cold gas distribution that increases the scale-length of Ly$\alpha$ (Zheng et al. 2011). In the future, deep Ly$\alpha$ observations of protoclusters may thus illuminate the infalling gas on large scales, the in- and outflowing gas close to the central sources, and map the small-scale gas distribution between galaxies in overdense regions (see Sect. 6.2).

5.2 Protoclusters associated with radio galaxies and QSOs

We have discussed some of the evidence for protoclusters that have been found by targeting radio galaxies and QSOs, but also seen evidence of spectacular examples of protoclusters that do not, or at least are not known to, contain any particularly powerful AGN. This raises a number of interesting questions. In Sect. 5.2.1 we will first discuss the protocluster environments of radio galaxies and QSOs in general. In Sect. 5.2.2 we will look more closely at the environments of QSOs at $z\sim 6$ given the strong interest in this area, specifically.

5.2.1 The environments of radio galaxies and QSOs

Based on the literature overview presented in Sect. 3, a significant fraction of known protoclusters was found by targeting radio galaxies, suggesting a deeper connection between the two. In order to get some idea of the statistics on the occurrence of protoclusters near radio galaxies, it is best to look at studies that targeted large samples. Unfortunately, there are only few of such studies to date. Venemans et al. (2007) found that at least 75 % of radio galaxies are in protoclusters traced by overdensities of LAEs, based on their sample of 9 radio galaxies ($z\approx 2-5$ and $L_{2.7GHz}>10^{33}\,\hbox {erg} \hbox {s}^{-1}\, \hbox {Hz}^{-1}$). The evidence for protoclusters around these systems was primarily based on large overdensities of spectroscopically confirmed LAEs, but has since been corroborated by other galaxy populations in a number of cases. More evidence for protoclusters around radio galaxies has been uncovered since the seminal work of Venemans et al. (2007), such as some newly identified protoclusters around other radio galaxies (Matsuda et al. 2009; Hatch et al. 2011b; Hayashi et al. 2012; Cooke et al. 2014; Shimakawa et al. 2014) and a number of much larger statistical studies (e.g., Hatch et al. 2011a; Galametz et al. 2012; Wylezalek et al. 2013).

Hatch et al. (2011a) performed a near-IR selection of galaxies in six radio galaxy fields at $z\sim 2.4$, finding three fields to be overdense by a factor of ${\sim }3\,(3\sigma$ deviations from the field surface density on scales of $R<3$ comoving Mpc), consistent with protoclusters (Hatch et al. 2014). Although the study used color selections that have considerable redshift uncertainty, one of the three fields was later shown to be overdense in narrow-band selected HAEs as well (Cooke et al. 2014). Galametz et al. (2012) observed 48 radio galaxies at $1.2<z<3$ using Spitzer and measured the surface densities of galaxies selected in the mid-IR. They found that 73% of their fields was denser than the median field survey, while 23% was denser than the field at the $\gtrsim 2\sigma$ level. Galametz et al. (2012) and Hatch et al. (2014) also analyzed trends between overdensity and redshift or radio luminosity, finding no correlations. The CARLA survey targeted 387 radio-loud AGN at $1.3<z<3.2$ (187 radio-loud QSOs and 200 radio galaxies; Wylezalek et al. 2013). The results indicate that 55% (10%) of the AGN fields are overdense at the $2\sigma \, (5\sigma$) level. No difference was found between the environments of the radio galaxies and the radio-loud QSOs, consistent with the idea that orientation is the only distinguishing parameter that separates these two populations. If we optimistically assume that about half of the radio AGN studied by Wylezalek et al. (2013) indeed lie in $2-5\sigma$ overdensities, this would imply that the other half of the sample occupies much more average and even underdense regions. However, it is not clear whether the fields in which no or only small overdensities were detected indeed do not contain protoclusters. The study recovered several previously known overdensities (clusters and protoclusters), while their color-selection also proved relatively insensitive to even some known protoclusters in these fields. Another complication arises from the fact that the tracer galaxies (e.g., radio galaxies or QSOs) are usually assumed to lie at or near the center of their (proto-)cluster. Many of the fields show substantial sub-clustering, and fields that do not show strong central overdensities close to the radio source often still show an excess just a few arcminutes away (Hatch et al. 2014). Although these studies were the first to measure the environments for a very large number of high redshift radio sources statistically, they naturally lack the redshift precision of, e.g., Venemans et al. (2007) due to the limitations inherent to the mid-IR color selection (Falder et al. 2011). A definite answer to the question what fraction of radio sources are in (proto-)clusters must therefore await future work.

Many searches for protoclusters by specifically targeting QSOs have also been performed (e.g., Campos et al. 1999; Djorgovski et al. 2003; Wold et al. 2003; Kashikawa et al. 2007; Falder et al. 2011; Husband et al. 2013; Adams et al. 2015; Hennawi et al. 2015). A large statistical survey very similar to the Spitzer study of Wylezalek et al. (2013) was performed by Falder et al. (2011). They analyzed the environments of 46 luminous ($M_i\le -26$), predominantly radio-quiet, QSOs. They also found significant overdensities based on the surface densities of sources around QSO subsamples centered at $z\sim 2\, ({>}4\sigma$) and $z\sim 3.3\,({>}2\sigma$), interpreted as evidence for protoclusters similar to Wylezalek et al. (2013). However, several recent works based on spectroscopic observations of LBGs and LAEs more similar to the Venemans et al. (2007) work on radio galaxy environments, have found evidence for protoclusters near luminous QSOs in about 10% of the cases (Trainor and Steidel 2012; Adams et al. 2015; Hennawi et al. 2015), although in at least one case the overdensity is one of the largest known (Hennawi et al. 2015). These results suggest that the success rate of protocluster discovery may be lower when using QSOs compared to radio galaxies.

How can we explain this apparent discrepancy between the (protocluster) environments of radio galaxies and (radio-quiet) QSOs? Luminous QSOs and radio galaxies at high redshift are strongly clustered sources in average to massive dark matter halos (e.g., Shen et al. 2007, 2009; Overzier et al. 2003; Donoso et al. 2010; Falder et al. 2011; White et al. 2012; Eftekharzadeh et al. 2015). At $z\sim 2-3$, typical optical QSOs live in halos of $M_\mathrm{h}\sim 2\times 10^{12} h^{-1} M_\odot$ (White et al. 2012; Trainor and Steidel 2012; Eftekharzadeh et al. 2015), with no measurable dependence on redshift or luminosity. The halos of these QSOs will evolve into halos of ${\sim }10^{13} h^{-1} M_\odot$ today, which is an order of magnitude lower than those expected if these QSOs traced protoclusters at $z\sim 3$ (Chiang et al. 2013 and Fig. 9). Because of short duty cycles and the lack of correlation between luminosity and clustering, QSOs in general tend to trace average halos, regardless of luminosity or black hole mass (Font-Ribera et al. 2013; Eftekharzadeh et al. 2015; Orsi et al. 2016). This explains, at least in part, the results of the many different QSO environment studies. However, Shen et al. (2007) found that the most luminous QSOs at $3<z<4.5$ have a much larger correlation length that is comparable to that of massive clusters today (Postman et al. 1992). The (minimum and median) masses inferred are an order of magnitude higher than those of the earlier sample ($M_h\sim 4-6\times 10^{12} h^{-1} M_\odot$ at $z\sim 4$), and in the range of those expected for the central halos of protoclusters at these redshifts (Fig. 9). As argued by Adams et al. (2015), these QSOs should have a high duty cycle (${\gtrsim }10$%) and lie on a relatively tight relation between QSO luminosity and halo mass (see Martini and Weinberg 2001; Shankar et al. 2010), in contrast to the more typical optical QSOs at these and lower redshifts (see also McGreer et al. 2016). Radio-loud QSOs at $z\lesssim 2$ are furthermore more strongly clustered than radio-quiet QSOs matched in luminosity or SMBH mass ($M_h\sim 10^{13} h^{-1} M_\odot$; Shen et al. 2009), and radio galaxies at $z\sim 1.5$ have $M_h\sim 10^{13.6} M_\odot$ (Allison et al. 2015). If this correlation between characteristic halo mass and radio-loudness extends to higher redshifts, we would expect that protoclusters are indeed found more frequently near radio galaxies relative to QSOs.

These results are consistent with predictions from semi-analytical models (e.g., Fanidakis et al. 2011, 2013; Orsi et al. 2016). Radio galaxies are typically hosted by the most massive haloes present at any redshift, and their typical $z=0$ descendant halo is more massive than that of the halos hosting typical QSOs. In these models, the difference between radio galaxy and QSO hosts mainly comes about because the radio power is assumed to scale with black hole mass and spin, both of which are larger in more massive halos. Radio galaxies furthermore rely on a low accretion rate regime to allow the formation of an advection dominated accretion flow that fosters radio jets, while QSOs are assumed to form in high accretion rate regions that do not lead to strong jets (e.g., see Fanidakis et al. 2011, 2013; Hatch et al. 2014; Orsi et al. 2016). Radio sources at $z>1$ are found in denser-than-average structures compared to ordinary galaxies of the same stellar mass, indicating that radio activity may indeed be preferentially triggered in dense environments (see also Rawlings and Jarvis 2004). The triggering of radio jets requires both the presence of a rapidly spinning SMBH with $M_\mathrm{BH}\gtrsim 10^8 M_\odot$ (McLure and Jarvis 2004; Fanidakis et al. 2011) as well as the inflow of gas. Hatch et al. (2014) suggest that these requirements are most easily met in the gas-rich cores of protoclusters where frequent mergers between massive galaxies furthermore help to increase the mass and spin of the SMBHs (see also Chiaberge et al. 2015).

5.2.2 The environments of QSOs at $z\sim 6$

The luminous QSOs at $z\sim 6$ are believed to be associated with dense matter peaks in the early universe. Their very low space density of about $1\,\hbox {per}\,\hbox {Gpc}^3$ implies a (maximum) host halo mass of ${\sim }10^{13} M_\odot$ (e.g., Fan et al. 2001; Springel et al. 2005; Li et al. 2007). A halo that massive at $z\sim 6$ will easily evolve into a cluster-sized halo by the present-day, even if it experiences only modest (say, a factor of 10) growth. Based on the expectation that these extreme high redshift objects should pinpoint the locations where massive clusters of galaxies are forming, a significant amount of attention has recently been devoted to studying the environments of luminous QSOs at $z\sim 6$.

To date, the environments of $\approx 13$ QSOs between $z=5.7$ and $z=7.1$ have been investigated, searching for faint LBGs in deep pencil-beam surveys with HST (Stiavelli et al. 2005; Zheng et al. 2006; Kim et al. 2009; McGreer et al. 2014), or for bright LBGs (Willott et al. 2005; Utsumi et al. 2010; Simpson et al. 2014; Morselli et al. 2014) and LAEs (Bañados et al. 2013; Mazzucchelli 2016) in wider field ground-based data (see Mazzucchelli 2016, for an overview). Willott et al. (2005) found no evidence for excesses of $z^\prime <25.5$ LBGs within ${\sim }10$ Mpc (co-moving) fields around 3 QSOs, while Utsumi et al. (2010) and Morselli et al. (2014) found excesses within much wider ${\sim }$50 (co-moving) Mpc fields around 5 QSOs [including those of Willott et al. (2005)]. Some QSOs show an excess of fainter LBGs detected with HST (Stiavelli et al. 2005; Zheng et al. 2006), but these appear to be not as common as QSOs in relatively underdense or average density regions (Kim et al. 2009; McGreer et al. 2014; Simpson et al. 2014). Perhaps the strongest constraints on QSO environments at these redshifts come from narrow-band observations, which did not find any excess of LAEs within $\Delta z\approx 0.1$ for 2 QSOs (Bañados et al. 2013; Mazzucchelli 2016).

Summarizing these results, there is currently very little observational evidence that supports the conclusion that the $z\sim 6$ QSOs are in large overdense regions or protoclusters. We note that many studies relied on the dropout technique to isolate $z\sim 6$ galaxies in the vicinity of the QSOs. Although this technique works reliably for such galaxies in the field, it is sensitive to galaxies from a wide redshift interval ($\Delta z\sim 0.6$). If the QSOs typically trace large overdense regions, we may still have expected to see a statistical excess of dropout galaxies despite the relatively crude redshift selection, but this does not appear to be the case for most fields observed. On the other hand, for QSO fields that are average or slightly over- or underdense, we cannot rule out that deeper data or spectroscopic observations may still reveal significant overdensities (see Husband et al. 2013, for such cases at $z\sim 5$). Although a full census of the QSO environments will need to await future studies, we can, however, already state with certainty that there exist highly overdense regions at $z\sim 6$ that display much larger concentrations of galaxies than those observed near QSOs (e.g., Ouchi et al. 2005; Toshikawa 2014; Ishigaki et al. 2016). For example, the $z=6.01$ protocluster studied by Toshikawa et al. (2012); Toshikawa (2014) is one of the largest concentrations of galaxies found at these redshifts, and such a structure would have easily been detected if it existed around any of the $z\sim 6$ QSOs previously studied (Fig. 17).

Fig. 17

A protocluster at $z\approx 6.01$ in the Subaru Deep Field (SDF; Toshikawa et al. 2012; Toshikawa 2014). Red solid circles mark the positions of the 10 spectroscopically confirmed protocluster members, while blue triangles mark other $z\sim 6$ galaxies. i-dropouts without spectroscopic confirmation of Ly$\alpha$ are indicated by the green squares. Contours indicate the surface density of i-dropouts at 0–6$\sigma$. The large excess and significant clustering of Ly$\alpha$ and dropout galaxies in the SDF protocluster are stronger than that observed near any of the $z\sim 6$ QSOs studied to date. Figure reproduced from Fig.1 in Toshikawa (2014)

Both the observations described above and simulations (Overzier et al. 2009a; Romano-Diaz et al. 2011) suggest that the environments of QSOs observed are inconsistent with these sources tracing the richest regions of the early universe, in stark contrast with the simple theoretical expectations that motivated many of these studies. It is unclear what causes this discrepancy. In principle, it is still possible that current observations failed to detect significant overdensities because they do not reach a sufficient depth or area, or because the galaxies in the QSO environment are highly obscured by dust. However, this would be somewhat surprising given that several highly significant (non-QSO) protoclusters were found at $z\sim 6$ using data of very similar quality as that obtained for the QSO fields. Also, no dusty star-forming galaxies have been found in early Atacama Large Millimeter Array (ALMA) observations of $z>6$ QSOs (Venemans et al. 2016). Other authors have suggested that strong radiative feedback may suppress galaxy formation in QSO environments (e.g., Kashikawa et al. 2007; Utsumi et al. 2010), but it is not clear if QSO feedback could offset gas cooling uniformly in all directions and on scales as large as the full extent of protocluster regions (several tens of Mpc at $z\sim 6$). An alternative option that is consistent with most of the data, is that the $z\sim 6$ QSOs may trace regions that, on average, are significantly less overdense compared to the theoretical expectation.

Simulations show that there is a non-trivial relation between the halo mass at $z\sim 6$ and that of its descendent at $z=0$ (e.g., Overzier et al. 2009a; Romano-Diaz et al. 2011; Angulo et al. 2012). Angulo et al. (2012) showed that there is a good linear correlation between the overdensity at $z=6$ and the halo mass at $z=0$ that is stronger than the correlation between halo mass at the two redshifts. Furthermore, the slope of the relation between overdensity and $z=0$ mass is steeper and has a smaller scatter when the density is determined on large smoothing scales (${\sim }$7–28 Mpc) compared to small smoothing scales (< few Mpc). This suggests that it is the large-scale environment of a $z\sim 6$ QSO that determines whether a massive cluster will develop or not. These results may very well explain the observations of QSO environments at these redshifts. Some important guidelines that can be deduced are the following. The redshift evolution of massive dark matter halos between $z\approx 6$ and $z\approx 0$ is not as straightforward as often assumed, and the most massive halo at $z\sim 6$ will typically not be the most massive halo at $z=0$ (Angulo et al. 2012). This could be the case for some of the early protoclusters detected (Ouchi et al. 2005; Toshikawa 2014; Ishigaki et al. 2016). Also, there is a non-zero possibility that QSO-hosting halos of relatively low mass at $z\sim 6$ will still become among the most massive halos by $z=0$. In this scenario, the QSOs could be in average or mildly overdense environments at $z\sim 6$ and still end up near the top of the local $M_\bullet -\sigma _*$ relation (McConnell et al. 2012). In any case, we expect a large scatter between the overdensity of halos measured at $z\sim 6$ and the present-day descendant halo mass. Overdensities measured on very large (${\gtrsim }7$ Mpc) scales are a better indicator than those measured on small scales (Angulo et al. 2012), and it will thus be important to obtain sufficiently deep observations over very large fields of view before we can rule out that $z\sim 6$ QSOs end up in massive clusters today.

It will be interesting to see a definitive conclusion about the extreme (or not) environments of the first luminous QSOs at $z\gtrsim 6$. Deep narrow-band studies are limited to a number of atmospheric windows where the OH night-sky emission is low. Bañados et al. (2013) therefore suggest to specifically search for QSOs at $z\approx 5.7, z\approx 6.6$ and $z\approx 7.0$, and then target their environments in the narrow redshift intervals of the filter. In light of the Angulo et al. (2012) predictions, it will be important to search for LBGs and LAEs in very wide fields around these QSOs, while ALMA could be used to search for the far-infrared line emission from obscured galaxies at the redshifts of the QSOs.

5.3 Protoclusters and the epoch of reionization

The epoch of reionization (EoR), during which hydrogen in the universe transitioned from being mostly neutral to being mostly ionized, is an extremely important event for cosmology, structure formation and galaxy evolution (e.g., see Mellema et al. 2013, for a review). Although in principle there are a number of candidates for the sources responsible for reionization, such as individual stars, galaxies and (mini-)quasars, it seems likely that the stars of the first generation of galaxies were responsible for producing the bulk of the Lyman continuum photons that ionized the neutral IGM (e.g., see Robertson et al. 2015; Bouwens et al. 2015b).

Simulations of structure formation show that as early as $z>6$, the universe contains significant density fluctuations on large scales (tens of Mpc, see Fig. 1). The early ionizing radiation field was dominated by the combined star formation of galaxies in these large overdense regions. Once the ionized bubbles around the large numbers of individual galaxies in these regions made contact, large ionized regions measuring tens of Mpc in diameter formed surrounded by a sea of neutral hydrogen. As reionization proceeded, the ionization front penetrated from the high into the surrounding lower density regions in the so-called “inside-out” fashion. It is thus expected that early density fluctuations will lead to a patchy reionization on large scales. An example showing the differences expected between field and protocluster regions is shown in Fig. 18, reproduced here from Ciardi et al. (2003). Initially, the high photon production rate in large overdensities causes completely ionized regions around protoclusters that are both larger and that have an overall lower neutral fraction compared to the field at similar redshifts (left and middle panels of Fig. 18). At late times, however, the cores of protoclusters may be among the last regions to reionize given the high recombination rates in these very dense regions (right panels). Large simulations performed by Iliev et al. (2014) also find that inside-out reionization leads to large voids that remain neutral longer compared to the high-density peaks and filaments. These differences will be reflected in the fluctuations in the redshifted 21 cm emission line signal used to probe this epoch. The redshift at which a certain ionized fraction ($f_\mathrm{HII}$) is reached can differ substantially depending on whether a region is under- or overdense with respect to the mean density. Iliev et al. (2014) find that for a fixed value of $f_\mathrm{HII}$, the redshifts at which those ionized fractions are obtained differ by as much as $\pm 0.04, \pm 0.2$ and $\pm 0.5$ when measured on scales of ${\sim }$200, 100 and 50 Mpc. The mean redshift at which the typical ${\sim }50$ Mpc-size region reaches an ionized fraction of 10% (90%) is $z\approx 8.6\,(z\approx 6.9$), whereas these redshifts are $z\approx 9.1\, (z\approx 7.3$) for regions that are 10% overdense compared to the mean density (see Fig. 13 in Iliev et al. 2014).

Fig. 18

Simulated maps of the neutral and ionized IGM during the EoR in field (top panels) and protocluster regions (bottom panels). Figure reproduced from Fig. 4 in Ciardi et al. (2003)

Cen (2005) have suggested that the ionizing photons produced by the progenitors of the faint dwarf galaxies in present-day clusters may have been responsible for a large fraction of the photons required for reionization. In fact, the study showed that protoclusters could be capable of fully ionizing the IGM entirely by themselves. Because the earliest HII regions will form in large overdense regions that are highly biased and strongly clustered, the overlapping of protocluster-sized regions will create large ionized regions and large spatial variations in maps of the neutral gas fraction. It is thus highly plausible that protoclusters of galaxies played an important, perhaps even dominant role in the process of reionization. Upcoming reionization measurements may be able to see these protoclusters, first as the very large-scale overdensities of neutral hydrogen before reionization starts, then as the first very large patches of relatively low neutral hydrogen fraction. In some sense, reionization studies will be a study of cluster formation.

It is interesting to note that the very rare and brightest dropout galaxies seen toward the end of reionization near $z\sim 6-10$ have number densities of ${\lesssim }10^{-6}\,\hbox {Mpc}^{-3}$, i.e., equal to those of rich clusters today (Bouwens et al. 2016). It would be interesting to know whether this is just a coincidence, or whether these highly biased galaxies can only exist because they formed as part of the large overdense regions associated with protoclusters. Ono et al. (2012) has found an indication that the fraction of Ly$\alpha$ emitting galaxies drops more strongly between $z\sim 7$ and $z\sim 6$ in UV-faint galaxies compared to UV-bright galaxies. This suggests that the IGM, which is believed to be responsible for the absorption of Ly$\alpha$ in star-forming galaxies, became more neutral toward higher redshift but the change is not as pronounced for brighter galaxies. This is consistent with the inside-out reionization model if the brighter LAEs trace higher density regions in which reionization was completed earlier compared to the lower density regions traced by the fainter galaxies.

Besides these direct links between protoclusters and the EoR, there is yet another way in which protoclusters can play an important role. Lyman continuum observations of high redshift galaxies can give important clues to the processes responsible for the escape of ionizing photons. Protoclusters contain large numbers of star-forming galaxies that are relatively bright in the rest-frame far-UV (LBGs and LAEs). Because these galaxies have a narrow redshift distribution and occupy a relatively compact area on the sky, protoclusters are good targets for studying the escape of ionizing photons based on deep imaging or multi-object spectroscopic observations below the Lyman limit (see Mostardi et al. 2013, for an example using a protocluster at $z=2.85$).

6 Outlook

6.1 Future observations

We have seen in this review that there exist excellent examples of protoclusters over a wide range in redshift and with a variety of distinguishing properties that have been used to shed light on important aspects of cluster formation prior to $z\sim 2$. However, if we want to observe this process from its earliest stages to the present, we will need to obtain statistical and unbiased samples of protoclusters in various bins of redshift, mass and evolutionary stage, rather than the highly heterogeneous and relatively limited data set we have available today. Fortunately, there are a number of new instruments and surveys that should soon be able to provide us for the first time with the large samples needed for such an analysis. Because protoclusters are rare and high redshift galaxies are faint, protocluster science should benefit enormously from deep, wide optical and near-infrared imaging surveys. Surveys with a high discovery potential are the Kilo-Degree Survey (KiDS; 1500 $\hbox {deg}^2$ in ugri; de Jong et al. 2013), the Dark Energy Survey (DES; 5000 $\hbox {deg}^2$ in grizY; Diehl et al. 2014) and the Hyper Suprime-Cam Wide survey (HSC-Wide; 1400 $\hbox {deg}^2$ in grizY; Takada et al. 2014). Due to its greater depth, HSC-Wide will be the most promising for finding large numbers of protoclusters based on the detection of surface overdensities of high redshift dropout galaxies. In a recent pilot project based on CFHTLS data, Toshikawa et al. (2016) found that about 80% of photometrically selected $\delta >4\sigma$ galaxy overdensities correspond to actual protoclusters of massive present-day clusters. The number density of such regions is about 1 $\hbox {deg}^{-2}$ per unit redshift from $z\sim 3$ to $z\sim 6$, which translates into more than 1000 high quality systems per unit redshift expected over the whole HSC-Wide area. Much smaller area (a few to a few tens of $\hbox {deg}^2$) but significantly deeper components of the HSC survey will include several narrow-band filters that will be used to perform a blind search for protoclusters using dropout galaxies and LAEs at specific, relatively narrow redshift intervals. The greater depth and efficient redshift selection will allow us to, e.g, compare UV and Ly$\alpha$ luminosity functions in protoclusters and in the field, determine the Ly$\alpha$ escape fraction as a function of overdensity, determine the topology and substructure of protocluster regions, search for correlations between overdensity, quasar activity and diffuse Ly$\alpha$ emission (halos or LABs), or probe the distribution and metal abundance of the gas in the infall regions around protoclusters detected in absorption against background QSOs and galaxies. The Large Synoptic Survey Telescope (LSST) will probe an even larger area and to a greater depth compared to HSC-Wide (20000 $\hbox {deg}^2$ in ugrizY). Combining these deep optical surveys with matched observations in the 1–5 $\mu$m range will improve photometric redshifts and allow the selection of dusty star-forming galaxies and passively evolving galaxies at $z\simeq 1-3$ and is therefore ideal to select and study high redshift (proto)cluster candidates and their stellar populations. The Wide Field Infrared Space Telescope (WFIRST) will perform large area imaging and spectroscopy in the near-IR to unprecedented depth that will enable us to both study the assembly of galaxy clusters and their galaxies at $z\simeq 1-3$, and to obtain large statistical samples of galaxies, quasars and protoclusters during the important first billion years of cosmic history.

Perhaps the highest potential for protocluster studies will be to sample the cosmic web at high redshift with high spectroscopic completeness. If the volumes are large enough, protoclusters will naturally appear in the catalogs extracted from these surveys without having to make any pre-selections or guesses about what constitutes a protocluster or where to find them. Also, such surveys will be able to immediately provide detailed physical properties related to the kinematics, stellar populations and gas properties of galaxies inside and outside the overdense regions, provided that the spectra have sufficient signal-to-noise and spectral resolution (e.g., Steidel et al. 2000, 2005; Cucciati et al. 2014; Lemaux et al. 2014). New large multiplexed fiber-fed spectrographs that are being planned for 4–10 m telescopes will be extremely suitable for extending such studies to larger areas of the sky and a larger redshift range.

For example, the Hobby–Eberly Telescope Dark Energy Experiment (HETDEX; Hill et al. 2008) is a blind $450\,\hbox {deg}^2$ spectroscopic survey sensitive to LAEs at $1.9< z < 3.5$ that offers a strong advantage for finding homogeneous samples of protoclusters compared to standard broad- and narrowband imaging surveys (Chiang et al. 2013; Orsi et al. 2016). Chiang et al. (2015) showed that HETDEX data will be sufficient to estimate the $z=0$ cluster masses corresponding to the overdensities of LAEs in protoclusters. HETDEX will furthermore detect quasars, radio galaxies and LABs, providing relatively unbiased and statistically meaningful data on the relation between such objects and protoclusters. Other advances will come from the next generation of highly multiplexed fiber spectrographs. The William Herschel Telescope Enhanced Area Velocity Explorer (WEAVE; Dalton et al. 2012) and the 4 m Multi Object Spectroscopic Telescope (4MOST; Jong et al. 2014) will both have of order 1000 fibers over a few square degree field of view to perform medium resolution (MR) spectroscopy. The Multi-Object Optical and Near-infrared Spectrograph (MOONS; Cirasuolo et al. 2014) is a new ${\sim }1000$ fiber MR spectrograph planned for the Very Large Telescope (VLT), while the Prime Focus Spectrograph (PFS; Takada et al. 2014) for the Subaru Telescope will be capable of performing wide, deep spectroscopic surveys owing to its ${\sim }2400$ fibers that can be positioned across a 1.3-deg field. Surveys with such large multiplexing capacity will be able to detect high redshift star-forming galaxies (mostly LAEs and LBGs) with sub-Mpc (comoving) line of sight resolutions. At such a high resolution the measured overdensities of large-scale structures such as protoclusters will be very close to their true spherical overdensities, unlike narrow-band surveys that typically probe comoving depths of ${\sim }100$ Mpc (Chiang et al. 2013; Orsi et al. 2016).

Although the direct detection of 21 cm emission from individual galaxies with new radio facilities such as the Square Kilometer Array (SKA) and its pathfinders will be difficult at high redshift, protoclusters may still be discovered by searching for highly clustered star-forming galaxies and AGN detected spectroscopically through their lower-level CO transitions or, without redshifts, in the radio continuum. Very high redshift radio galaxies may be identified by pushing the ultra-steep spectrum technique that was used to discover many of the highest redshift radio sources known to date to even higher redshifts (Röttgering et al. 1997, 2006). Such distant $z>6$ radio sources would be very powerful tracers of protoclusters and SMBHs, and could also be used to study the IGM during the epoch of reionization by searching for 21 cm absorption along the line of sight (Ciardi et al. 2013).

We have already seen in Sect. 2.2.3 that IGM absorption techniques targeting background quasars and galaxies are extremely promising. The tomographic mapping of the cosmic web at high redshift relies on a dense packing of background sources, requiring high S/N spectra of star-forming galaxies rather than the much more luminous but much rarer QSOs. This requires 8m class telescopes and instruments such as MOONS and PFS. The IGM absorption studies will not only offer a new and competitive technique for finding galaxy protoclusters, they will also open up a new window to studying the local and bulk gas properties of these complex environments.

In Sect. 5.3, we have shown that the mapping of the distribution of ionized and neutral hydrogen during the EoR in some sense also maps out the distribution and properties of protoclusters, given that a significant, perhaps even dominant, fraction of all star formation at these redshifts is taking place in these highly biased, overdense regions. Upcoming surveys that trace the neutral or ionized gas distribution during this epoch such as the SKA will be able to test this.

It will remain challenging to increase significantly our capacity for detecting clusters of $M\gtrsim 10^{14} M_\odot$ at $z\gtrsim 2$ even with state-of-the-art X-ray missions (e.g., eROSITA), SZ effect experiments (e.g., SPTpol, ACTpol), and optical/near-IR surveys (e.g., LSST, EUCLID) (Weinberg et al. 2013). However, as shown in this review, protoclusters are typically identified through their characteristic large-scale overdensities of dark matter, galaxies or gas, and thus the technical limitations that affect classical cluster searches do not apply. A final argument for employing the enormous upcoming multi-wavelength survey capability is to provide the most suitable targets for detailed follow-up with facilities such as ALMA, the James Webb Space Telescope (JWST), and the planned 30-m class telescopes, as these facilities are not capable of performing large surveys.

6.2 Concluding remarks

The data assembled in this review demonstrate that the redshifts, sizes, galaxy and mass overdensities, collapse redshifts and descendant halo masses of many structures found in the $z\sim 2-8$ redshift range are consistent with those expected for the progenitors of galaxy clusters (Sects. 2 and 3). However, we must remain skeptical both when discussing the properties of single systems as well as ensemble properties, as the measurements are still affected by small number statistics, substantial uncertainties, incompleteness and selection bias.

The galaxies in these overdense regions will evolve from a protocluster-like regime that is relatively gas-rich, rapidly star-forming and has only weak environmental impact, to a cluster-like regime that is relatively gas-poor, quiescent and has stronger environmental impact (Sect. 4.1; Fig. 19). We have shown that the epoch between $z\sim 2.5$ and $z\sim 1.5$ should be a particularly important epoch for studying how exactly these transitions affect the properties of (proto)cluster galaxies and the ICM (see also the discussion in Valentino et al. 2015), and protoclusters may thus be crucial for confirming several key aspects of galaxy and structure formation models. We have shown evidence that red sequence galaxies are starting to appear in some protoclusters, possibly at a rate that is faster than that in the field (Sect. 4.2). However, with the exception of Lemaux et al. (2014) and Wang et al. (2016), few of these studies are based on spectroscopic evidence. New surveys such as VUDS are allowing for a more uniform selection of protocluster galaxies relatively independent of their star formation activity, although the spectroscopic confirmation still relies on the presence of emission lines or substantial flux in the UV part of the spectrum to identify breaks or absorption features. It is notoriously challenging to confirm spectroscopically truly “passive” galaxies at $z\gtrsim 2$ (Kriek et al. 2006, 2009; Toft et al. 2012), although this situation is improving (e.g., Kriek et al. 2015). Protoclusters also offer the unique opportunity to study the formation of BCGs (Sect. 4.3). Although the observational data are still rather limited to only a few well-studied systems, the growing number of protoclusters will eventually allow us to track the growth of BCGs more consistently across the $z\simeq 0-3$ redshift range in order to obtain an accurate picture of the SFHs and morphological evolution of the most massive galaxies in the universe and their SMBHs.

The luminous extended Ly$\alpha$ emission around radio galaxies, QSOs and LABs that is frequently seen in protoclusters appears to be related to the gas distribution around powerful AGN in massive overdense regions (Sect. 5.1). If so, their space density, luminosities, metal abundances and morphologies may hold important clues to the evolution of the IGM/ICM in these overdense regions. Zirm et al. (2009) showed that the halos around radio galaxies evolve strongly with redshift from $z\sim 4$ to $z\sim 1$, even though the AGN power remained the same. This indicates that it is primarily the surrounding medium that is evolving. As massive galaxies and their halos grow, a virial shock will create a quasi-static atmosphere of hot gas that may limit cooling and switch off pre-existing cold flows. We therefore expect to see a decrease in the luminosity and spatial extent of the extended Ly$\alpha$ emission, eventually disappearing altogether, as protoclusters transition into clusters (Fig. 19).

Fig. 19

The growth history of protoclusters and clusters, highlighting some important evolutionary characteristics that can be tested with upcoming observations. The models and data shown are the same as in Fig. 11 (see the caption of that figure for details)

It is currently still difficult to give a very precise answer to the question of whether protoclusters are preferentially found near radio galaxies or quasars and, if so, why (Sect. 5.2). Also, we must realize that the literature is likely to be biased, especially against the reporting of negative results. It is probably far more common for authors to publish new candidate protoclusters and spectroscopic confirmation of protoclusters than it is to publish non-detections. Although it is currently next to impossible to quantify its effect, this author suspects that for every positive detection there is likely to be at least one unpublished non-detection. This problem mainly affects the biased tracer studies, and is probably less of an issue in the large field-type surveys. We will also have to try to deal with the observational incompleteness. The fact that a particular observation produced no evidence of a protocluster around some target, does not mean that it is not present. Venemans et al. (2002) have shown that the total number abundance of radio galaxies, corrected for the typical radio synchrotron lifetimes, is consistent with every present-day BCG having gone through a radio galaxy phase at high redshift (see also Miley and Breuck 2008; Hatch et al. 2014). The fact that there is strong evidence for protoclusters around high redshift radio galaxies, combined with the fact that their hosts are good candidates of (proto)BCGs supports this conclusion. The fact that the synchrotron lifetimes are relatively short also suggests that, for every protocluster traced by a radio galaxy (or radio-loud QSO), there should be at least several protoclusters without a radio source at the same epochs. This appears to be supported, at least qualitatively, by the fact that the number density of radio-quiet protoclusters is much higher than that of radio-loud protoclusters (e.g., Steidel et al. 2005; Chiang et al. 2014; Planck Collaboration et al. 2015; Toshikawa et al. 2016), although a quantitative comparison must await future data from large blind surveys. It will also be interesting to see a definitive conclusion about the extreme (or not) environments of the luminous quasars at $z\sim 6$ (Sect. 5.2.2). Deep, wide, narrow-band surveys are most efficient in a number of atmospheric windows where the OH night-sky emission is low. Bañados et al. (2013) therefore suggest to specifically search for QSOs at $z\approx 5.7, z\approx 6.6$ and $z\approx 7.0$, and then target their environments in these narrow redshift intervals. For a complete census of their environments, these data should be complemented with ALMA in order to assess the possible contribution from obscured star-forming galaxies. At even higher redshifts, protoclusters may play an important role during the EoR (Sect. 5.3).

Until recently, the first stages of cluster evolution could only be explored indirectly, either through the fossil record in galaxy clusters or in cosmological simulations. However, starting with the first discoveries of protoclusters at $z\sim 3$ in the late 1990s (e.g., Pascarelle et al. 1996; Fevre et al. 1996; Steidel et al. 1998), we are finally beginning to catch a glimpse of this important epoch of cluster evolution. Although the existing observational sample is still relatively limited, this situation is rapidly changing owing to the next generation of wide and deep surveys that will ensure very large, uniform samples of protoclusters to be discovered (Sect. 6.1). In the next few years, we can expect to see the boundary between the studies of “clusters” and “protoclusters” erode even further, eventually disappearing altogether, as we reach deeper into the realm of the protoclusters.