Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

In 2013, we submitted a project proposal to the JSC Juelich asking for computational time to continue our project “The Small Scale Structure of the Universe”. We have been informed from Juelich that our project had received high ratings from the referees but, as the requested amount of computing time would be not available in Juelich, JSC suggested to move the whole project to HLRS where we got the requested time at Hermit. Unfortunately, the procedure of moving our project to a new computer and a different environment took substantially longer time than expected. In particular, we encountered some problems in the running and continuation of simulations started in Juelich. Because we could not run the planned simulations timely, these runs were done elsewhere. These complications resulted in some changes to the project plan.

First of all we concentrated ourselves on the early stage of cosmological evolution, the period of reionization. The role of reionization in regulating the formation of small scale structure has been recognized already more than two decades ago [9], however only recently new observational data and improved numerical methods allowed to study the influence of reionisation on the formation and evolution of dwarfs in detail. We performed a series of radiative transfer simulations to study the reionisation history.

In a second project we were running a set of simulations within the MaGICC (http://www2.mpia-hd.mpg.de/~stinson/magicc/, “Making Galaxies in a Cosmological Context”) galaxy formation project. To this end we used the physical model of the MaGICC project and the initial conditions of the CLUES project (http://www.clues-project.org/, “Constrained Local UniversE Simulations”) to create an accurate model of the Local Group of galaxies, including the Milky Way, Andromeda, M33 and the myriad of dwarf galaxies.

This report is organized as follows: In the first section we discuss the effect of low-mass ionizing source suppression on the 21-cm signal from the epoch of reionization. In the second session we discuss the CLUES simulations performed with the MaGICC galaxy formation model.

2 The Reionization Period

The first billion years of cosmic evolution remain the only period in the history of the universe still largely unconstrained by direct observations. While we now have fairly detailed data on the Cosmic Microwave Background originating from the last scattering surface at redshift \(z \sim 1100\) and a wealth of multi-wavelength observations at later times, z < 6, the intermediate period remains largely uncharted. A number of ongoing observational programs aim to provide observations of this epoch in e.g. high-z Ly-α [29], CMB secondary anisotropies [56] and redshifted 21-cm [16, 32, 42]. Improved observational constraints could provide a wealth of information about the nature of the first stars and galaxies, their properties, abundances and clustering, the timing and duration of the reionization transition and the complex physics driving in this process.

During photoionization the excess photon energy above the Lyman limit heats the gas to temperatures above 104 K. The temperature reached generally depends on the local level of the ionizing flux and its spectrum [47]. Typical values are \(T_{\mathrm{IGM}} = \mathrm{10,000}\text{\textendash }\mathrm{20,000}\) K, but it could be as high as \(\sim \) 40,000 K for hot (Pop. III) black-body spectrum. However, the hydrogen line cooling is highly efficient for T > 104 K, particularly at high redshifts, where the gas is denser on average, which would typically bring its temperature down to \(T_{\mathrm{IGM}} \sim 10^{4}\) K, and possibly somewhat below that due to the adiabatic cooling from the expansion of the Universe.

This increase of the IGM temperature caused by its photo-heating results in a corresponding increase of the local Jeans mass. The Jeans mass is based on the linear theory of cosmological perturbations and for 104 K gas it is roughly \(M_{\mathrm{J}} \sim 10^{9}M_{\odot }\) (with some redshift dependence) [19, 23, 46]. The actual galaxy mass under which the gas infall, and thus star formation, is suppressed differs somewhat from this instantaneous Jeans mass since the mass scale on which baryons succeed in collapsing out of the IGM along with the dark matter must be determined, even in linear theory, by integrating the differential equation for perturbation growth over time for the evolving IGM [11, 12, 46]. In reality, determining the minimum mass necessary for a halo collapsing inside an ionized and heated region to acquire its fair share of baryons which subsequently cool further to form stars is even more complicated. It depends on the detailed, non-linear, gas dynamics of the process and on radiative cooling. There is no single mass above which a collapsing halo retains all its gas, and below which the gas does not collapse with the dark matter. Instead, simulations show that the cooled gas fraction in halos decreases gradually with decreasing halo mass [8, 9, 38, 53].

The typical halo sizes at which this transition occurs as derived by these different studies also vary. Thoul and Weinberg [53] found that photoionization suppresses star formation in halos with circular velocities below \(\sim \) 30 km s−1, and decreases the cooled gas mass fraction in larger halos, with circular velocities up to \(\sim \) 50 km s−1. Navarro and Steinmetz [38] found that the cooled gas fraction is affected by photoionization even in larger galaxies, with circular velocities up to \(\sim \) 100–200 km s−1. On the other hand, Dijkstra et al. [8] recently showed, using the same method as [53], that at high redshifts the suppression is not as effective, and somewhat smaller galaxies can still retain some cooled gas. For simplicity, we assume that star formation is suppressed in halos with masses below 109 M ⊙ and not suppressed in larger halos, in rough agreement with the linear Jeans mass estimate for 104 K gas and the above dynamical studies.

2.1 The Simulations

We start by performing very high resolution N-body simulations of the formation of high-redshift structures. We use the CubeP3M N-body code [17]. For these simulations we used two computational volumes, 500 h−1 Mpc = 714 Mpc and 244 h1−1 Mpc = 349 Mpc, both chosen so as to be representative for the large-scale reionization patchiness [26]. The corresponding particle numbers, at 69123 and 40003 are chosen to ensure reliable halo identification down to 109 M ⊙ (with 25 and 40 particles, respectively for the two volumes). As discussed above, \(M_{halo} \sim 10^{9}\,M_{\odot }\) is roughly the Jeans mass for gas at temperature 104 K, typical for the post-reionization IGM. The unresolved halos can be added using a sub-grid model, as discussed in detail in [1]. This model provides the mean local halo abundance based on the cell density and here is used to include halos with masses 108 M ⊙ < M halo < 109 M ⊙ in both simulations.

The background cosmology is based on WMAP 5-year data combined with constraints from baryonic acoustic oscillations and high-redshift supernovae (\(\varOmega _{M} = 0.27,\varOmega _{\varLambda } = 0.73,h = 0.7,\varOmega _{b} = 0.044,\sigma _{8} = 0.8,n = 0.96\)). The linear power spectrum of density fluctuations was calculated with the code CAMB [31]. Initial conditions were generated using the Zel’dovich approximation at sufficiently high redshifts (z i = 150) to ensure against numerical artefacts [6].

The radiative transfer simulations are performed with our code C2-Ray (Conservative Causal Ray-Tracing) [35]. The method is explicitly photon-conserving in both space and time or individual sources and approximately (to a good approximation) photon-conserving for multiple sources, which ensures correct tracking of ionization fronts without loss of accuracy, independent of the spatial and time resolution, with corresponding great gains in efficiency. The code has been tested in detail against a number of exact analytical solutions [35], as well as in direct comparison with a number of other independent radiative transfer methods on a standardized set of benchmark problems [20, 24]. The ionizing radiation is ray-traced from every source to every grid cell using the short characteristics method, whereby the neutral column density between the source and a given cell is given by interpolation of the column densities of the previous cells which lie closer to the source, in addition to the neutral column density through the cell itself. The contribution of each source to the local photoionization rate of a given cell is first calculated independently, after which all contributions are added together and a non-equilibrium chemistry solver is used to calculate the resulting ionization state. Ordinarily, multiple sources contribute to the local photoionization rate of each cell. Changes in the rate modify the neutral fraction and thus the neutral column density, which in turn changes the photoionization rates themselves (since either more or less radiation reaches the cell). An iteration procedure is thus called for in order to converge to the correct, self-consistent solution.

The N-body simulations discussed above provide us with the spatial distribution of cosmological structures and their evolution in time. We then use this information as input to a full 3D radiative transfer simulations of the reionization history, as follows. We saved series of time-slices, both particle lists and halo catalogues from redshift 50 down to 6, uniformly spaced in time, every Δ t = 11. 53 Myr, a total of 82 slices. Simulating the transfer of ionizing radiation with the same grid resolution as the underlying N-body (fine grid of 13, 8243 and 80003, respectively for the two volumes) is still not feasible on current computers. We therefore use a SPH-style smoothing scheme using nearest neighbors to transform the data to lower resolution, with 3003 or 6003 cells for the 500 h−1 Mpc and 250 h−1 Mpc boxes and 5003 cells for the 244 h−1 Mpc box, for the radiative transfer simulations. We combine sources which fall into the same coarse cell, which reduces slightly the number of sources to be considered compared to the total number of halos.

We characterize our source efficiencies through a factor g γ. We assign to each an ionizing photon production rate per unit time, \(\dot{N}_{\gamma }\), proportional to the total mass in haloes within that cell, M as introduced in [21, 22, 25]:

$$\displaystyle{ \dot{N}_{\gamma } = \frac{g_{\gamma }M\varOmega _{b}} {\varOmega _{0}m_{p}} \left ( \frac{\varDelta t} {10\;\mathrm{Myr}}\right )\,, }$$
(1)

where m p is the proton mass and \(g_{\gamma } = f_{\mathrm{esc}}f_{\star }N_{\star }\left (\frac{10\;\mathrm{Myr}} {\varDelta t} \right )\) is an ionizing photon production efficiency parameter which includes the efficiency of converting gas into stars, f ∗, the ionizing photon escape fraction from the halo into the IGM, f esc and the number of ionizing photons produced per stellar atom, \(N_{\star }\), Δ t is the time between two snapshots from the N-body simulation. All simulations include an approximate treatment of Lyman-Limit absorber systems. During the early evolution the photon mean free path is set by the neutral patches and Lyman-Limit Systems are unimportant, while at late times they set a mean free path of several tens of Mpc [48]. In the current simulations we roughly model this by imposing a hard limit on the distance an ionizing photon can travel, set at 40 comoving Mpc.

We have performed series of radiative transfer simulations with varying underlying assumptions about the source efficiencies and the suppression conditions imposed on the low-mass sources. The ones performed here include two models, as follows:

  • Partially suppressed low-mass halos (pS):

    For this model, the low-mass halos (108 M ⊙ < M < 109 M ⊙) contribute to reionization at all times. In neutral regions, we assign them a higher efficiency of g γ = 7. 1. In ionized regions, these small galaxies are suppressed resulting in diminished efficiency, with g γ = 1. 7. This situation arises if star formation remains ongoing, but at a lower rate. Physically, this situation could arise if the fresh gas supply is cut off or diminished by the photo-heating of the surrounding gas, but a gas reservoir remains available for star formation in the galaxy itself. In this case the high-mass sources (M > 109 M ⊙) are assigned efficiency g γ = 1. 7.

  • Mass-dependent suppression of low-mass halos (gS):

    Instead of a sharp decrease in ionizing efficiency as in the previous case, we also consider the gradual suppression of sources in ionized regions. As before, high-mass sources are assigned g γ = 1. 7 everywhere, and low-mass ones have that same efficiency when in neutral regions. In ionized patches, the low-mass sources are suppressed in a linearly mass-dependent manner, with 108 M ⊙ fully suppressed and 109 M ⊙ not suppressed at all.

2.2 Specific Simulations Ran and Sample Results

The simulations ran here form part of a larger PRACE Tier-0 project (Project PRACE4LOFAR, 22+19M core-hours awarded in fifth and ninth call), which is mostly based on Curie (CEA, France), with some work done on clusters in Sweden and elsewhere under a separately-granted project. At HLRS we have ran 3 simulations. Two were ran on Hermit and were based on the 244 h−1 Mpc volume, with 2503 grid size, and used the ‘pS’ and ‘gs’ suppression models, respectively. The third simulation (currently incomplete, since the allocation ran out) is based on the 500 h−1 Mpc volume, with 6003 grid and is using the ‘pS’ suppression model. The Hermit simulations were ran on 16,384 cores (4096 MPI processes × 4 OpenMP threads). The Hornet simulation was started on 6144 cores, increasing to 12,288 and 18,432 cores as more sources formed, creating more computational work. This used 8 OpenMP threads due to higher memory footprint of the larger grid, and 768, 1536 and 2304 MPI processes, respectively. The three runs used approximately 360,000, 400,000 and 450,000 core-hours, respectively.

Once the runs were set up properly and got going there were no significant issues encountered, apart from the fairly long queues (requiring up to a week or more to get through). Both Hermit and Hornet were fairly easy to use, although the wiki help was confusing at times and probably could be organized better and more transparently. The code ran well and fairly efficiently and fast, particularly on Hornet.

On the other hand, the workspace mechanism used at HLRS is by far the most opaque and user-unfriendly we have ever encountered for the many years we have used supercomputers. Due to this we have lost our codes and setup several times, resulting in a significant waste of time and efforts, and much frustration. If such an extreme disk management policy is really required (all other centres we have used do not need it, so this is not obvious), at the very least it should be set up so that it gives some kind of automatic warning before workspaces will be deleted, and maybe a short grace period, so as to ensure no data or codes are lost. Such a (fairly minor) change should make the systems much more user-friendly without really impacting on performance or disk management significantly.

Some preliminary results from our simulations are shown in Fig. 1. This shows the evolution of the differential brightness temperature of the redshifted 21-cm line from neutral hydrogen as seen by a LOFAR-like experiment (assuming no noise and foregrounds). The full analysis of the results is currently being performed and will be published in series of papers, the first of which is currently being finalized (Dixon et al., in preparation). These simulations will form the core of a library of models for analyzing and interpreting the results from the LOFAR Epoch of Reionization Key Science Project.

Fig. 1
figure 1

Position-frequency slices from our 244 h−1 Mpc volume simulations. These slices illustrate the large-scale geometry of reionization and the significant local variations in reionization history for several of our simulations as seen at redshifted 21-cm line (shown is the differential brightness temperature in mK) smoothed with a 5 arcmin gaussian beam and 0.4 MHz (boxcar) bandwidth filter. The spatial scale is given in comoving Mpc. The models partly ran here are L3 (pS) and L4 (gs)

Full size image

3 MaGICC CLUES to Galaxy Formation and Cosmology

The “Making Galaxies in a Cosmological Context” (MaGICC) project [5, 51] formed a suite of isolated, hydro-dynamically simulated galaxies that match observed galaxy scaling relations over a large mass range, from dwarf galaxies to Milky Way analogues.

The CLUES-project (Constrained Local Universe Simulations, [13]) provides constrained simulations of the local universe designed to be used as a numerical laboratory of the current paradigm of cold dark matter cosmology. The Local Group and its environment is the most well observed region of the universe. Only in this unique environment can we study structure formation on scales as small as that of very low mass dwarf galaxies.

In this project we are combining the physical model of the MaGICC project with the initial conditions of CLUES to simulate an accurate model of the Local Group of galaxies, including the Milky Way, Andromeda, M33 and the myriad dwarf galaxies.

The simulations will be used for unprecedented analysis of the complex dark matter and gas-dynamical processes which govern the formation of galaxies. The predictions of these experiments can be easily compared with the detailed observations of our galactic neighborhood. By simulating a local environment, more constraints can be placed on our model than were available with an isolated suite of galaxies, such as the requirement to match luminosity function. Outstanding issues such as the missing satellite, cusp/core and “too big to fail” problem will be probed in search of a self-consistent solution.

Moving the detailed modeling of the galaxy formation physics that is used in isolated galaxies in the MaGICC project, to the larger volume of the Local Group provided by the CLUES initial conditions, requires significant computational expenditure, making massively parallel supercomputing facilities essential.

In this introduction to the MaGICC CLUES program, we use simulated galaxies from the well studied MaGICC program to compare with isolated galaxies from the new MagiCC CLUES simulations.

3.1 Simulations and the MagICC Model

The MagICC simulations have previously been shown to match a wide range of scaling relations including the Tully-Fisher, luminosity-size, mass-metallicity, and HI to stellar-mass relations at z = 0 [5]. The simulations also match the evolution of the stellar mass-halo mass relation [27, 51], as derived by abundance matching [37] and several relations at high redshift [40]. The simulations also expel sufficient metals to match local observations [44, 54] of OVI in the circum-galactic medium [5, 50].

Therefore, our first goal is to ensure that the isolated MagICC CLUES simulations retain similar properties as the MagICC simulations. In Table 1 we list properties of 12 galaxies from the MaGICC project [4, 51], which were zoomed-in regions of a total cosmological volume of side 68 Mpc. Both sets of galaxies use a \(\varLambda\) CDM cosmology with WMAP3 parameters, i.e. H 0 = 73 km s−1 Mpc−1, Ω m = 0. 24, \(\varOmega _{\varLambda } = 0.76\), Ω baryon = 0. 04 and \(\sigma _{8} = 0.76\).

The second set are from a single CLUES simulation. Again the zoom-in technique is used, this time together with observational data imposed as constraints on the initial conditions, in order to simulate a cosmological volume with structures similar to those most representative in our local universe. Several dark matter-only realizations are run until a Local Group analogue is found. Then this Local Group region is re-simulated with baryons and at a higher resolution. The CLUES simulation used also follows a WMAP3 cosmology.

The 12 MaGICC disk galaxies listed in Table 1 are separated into two sub-sets labelled as Milky-Way (MW) and irregular (Irr) type galaxies, although all are disc galaxies with stellar masses ranging from 1×108–5×1010M⊙. From the CLUES simulation we have selected the halos that satisfy the following conditions, in order to compare isolated galaxies with the MagiCC suit: (1) Not a sub-halo, (2) Mhalo > 4×1010M⊙. These integrate a sample of 10 well resolved isolated galaxies. Since this is a Local Group simulation, the three most massive galaxies are loose analogues of the Milky Way, M31 and M33, and the rest are isolated dwarf galaxies.

Halos in both simulations have been identified using Amiga’s Halo Finder (AHF; [28]). Halo masses are defined as the mass inside a sphere containing \(\varDelta _{\mathrm{vir}} \simeq 350\) times the cosmic background matter density at redshift z = 0.

Table 1 Properties of isolated simulated galaxies ordered by halo mass
Full size table

3.2 Simulated Galaxy Properties

3.2.1 Rotation Curves

Rotation curves of observed galaxies provide significantly more information regarding the angular momentum of galaxies than is contained within the Baryonic Tully-Fisher relation, allowing more stringent constraints on galaxy formation models; constraints that have not previously been applied to simulated galaxies. High resolution observations of HI velocities, combined with studies of the gas and stellar mass distributions, provide detailed information on how the different mass components are radially distributed in galaxies with a wide range of rotational velocities Vr (e.g. [3, 7, 10, 30, 41, 45]).

Figures 2 and 3 show in different symbols the gaseous, stellar, baryonic and total rotation curves of the MaGICC and M-CLUES simulated galaxies, respectively. These are measured at radii ranging from 0.7 kpc to 10 × h where h is the disc scale length. The rotational velocities are calculated using the mass within spherical radial shells in the expression \(V _{circ}(r) = \sqrt{(GM(r)/r)}\), where G is the gravitational constant.

Fig. 2
figure 2

The rotation curves of the 12 MaGICC disk galaxies. Different symbols represent the rotation values due to different mass components (triangles: cold gas; stars: stars; squares: all baryons; circles: total). Simulations reproduce the variety of observed rotation curves. Furthermore, like in observations, the features present in the baryonic curves are reflected in the total one

Full size image
Fig. 3
figure 3

The rotation curves of the ten galaxies selected from the CLUES simulation. They reach lower circular velocities than the MaGICC sample

Full size image

The simulated galaxies reach a flat value of the velocity which persists to large radii, and they lack the strong peak at small radii that not so long ago was ubiquitous in simulations due to overcooling. A couple of galaxies, g5665_MW and MC1, have significant bulges, which is reflected in the heightened inner region of their rotation curves. Differences between the two sets of simulations are not evident, with galaxies of similar masses reaching similar maximum, flat velocities (see for example g7124_MW & MC3, g15807_Irr & MC1 or g5664_Irr & MC10).

3.2.2 Stellar Mass-Halo Mass

In the right panel of Fig. 4 the stellar-to-halo mass relation is plotted, along with the empirical relation [15, 36]. The MagICC simulations actually were tuned to match the stellar mass halo mass relation at one galaxy mass, and shown to then match the relation over a range of masses ([5, 40] and also the Nihao simulations [55] which use very similar implementation of physics as the MaGICC runs). The CLUES simulation were also calibrated, from experience, to match the relation. So, although in some sense it is not surprising that the simulations match the relation to which they were tuned, they actually match the relation over a far wider mass range than the one on which the parameter search was performed.

Fig. 4
figure 4

Left panel: The baryonic Tully-Fisher relation. Total baryonic mass Mb (stars + cold gas) plotted against circular velocity Vflat. Blue points are from the MagiCC suite while the red points are the CLUES simulations, with the line showing linear fit, with slope = 3.76. The small green points show Vmax rather than Vflat, which results in a slightly flatter relation, slope = 3.43, and slightly larger scatter (see text for details). Right panel: The stellar-to-halo mass relation. Also shown are the empirical stellar-halo mass relations of [15] (green line) and [36]. The MagICC galaxies are in blue while the CLUES simulations are in red

Full size image

3.2.3 The Baryonic Tully-Fisher Relation

In the left panel of Fig. 4 we plot the Baryonic Tully-Fisher relation, that is the mass of baryons of each simulated galaxies as a function of their circular velocity. There has been significant progress over the past years in our ability to simulate these processes of disc formation within a cosmological context. Without an efficient feedback scheme, angular momentum is lost to dynamical friction during the mergers of overly dense sub-structures (e.g. [33, 39, 43]). Progress was made by implementing increasingly effective recipes for feedback from supernovae [49, 52] and the inclusion of other forms of feedback from massive stars [18, 51]. The benchmark for assessing this progress has primarily been the ability to match the Tully-Fisher relation (e.g. [14]), with recent simulations matching the relation, and in particular the Baryonic Tully Fisher relation (BTFR), for galaxies over a range of masses [2, 5]. The latest simulations shown in Fig. 4 show perhaps the best reproduction of the BTFR for simulated galaxies to date.

The maximum circular velocity found in each simulated galaxy, Vmax, is a good approximation of the flat velocity, Vflat, in most cases. In the cases mentioned above where a couple of MW type galaxies have significant bulges, we show different values of Vmax and Vflat in Table 1. In the left panel of Fig. 4 we plot the BTFR using Vflat, with the MaGICC and M-CLUES sets of simulations shown as blue and red dots, respectively. Here the baryonic mass is defined as the sum of the mass coming from stars and cold gas particles, where the latter are estimated as a multiple of the atomic HI gas mass Mg = ηMHI, with \(\eta = 4/3\) (following e.g. [34]). In the case of MC7, the galaxy is about to undergo a merger, and we use the maximum velocity from the inner 10 kpc as Vflat which is the central galaxy, and use the baryonic mass from within this same radius. The scatter is very small, with the galaxy that is furthest from the fit being MC7, the one which has a very close companion galaxy with which it is dynamically interacting.

If we simply use Vmax in each case, the relation is slightly flatter, and can be seen as small green dots in Fig. 4, with a slightly larger scatter than in the case of Vflat. These relations are consistent with the observational fits found in the literature (see [34] for a summary), as is the trend for a flatter relation with greater scatter when using Vmax rather than Vflat.

4 Summary

After a significant delay in our project due to problems in the running and continuation of simulations started in Juelich we defined two new projects within our SSSU project at HLRS, namely radiative transfer simulations to study the reionisation history and simulating the formation of the Local Group using the physical model of the MaGICC project. Besides some problems with automatically deleted workspaces and the resulting delay in the project the simulations went very well at Hermit and Hornet. The preliminary results presented in the two sections of this report are not yet published. We plan to submit the corresponding papers by the end of the year.