1 Introduction

Anthropogenic emissions of CO2 and other greenhouse gases will provoke significant changes of the climate from the next decades to centuries. Substantial consequences for society, economy, and agriculture can be expected demonstrating the need of mitigation and adaptation. Although climate change is a global phenomenon, it will reveal its impacts on the regional scale. Mitigation, adaptation, and decicion making require robust climate data on high resolution in space and time. Therefore the World Climate Research Program initiated the COordinated Regional climate Downscaling EXperiment CORDEX [8] to numerically downscale the coarse resolution global coupled ocean-atmosphere general circulation models (GCMs) (grid cells > 120 km) analyzed in the IPCC (Intergovernmental Panel on Climate Change) report 2013 (http://www.ipcc.ch) with regional climate models (RCMs) to 50 km and 12 km resolution, respectively. These climate projections are typically run until 2100 and are following the so-called Representative Concentration Pathways (RCPs) for greenhouse gas emission scenarios.

Climate projections require an ensemble of simulations to assess not only the climate change signals, extreme values, and their uncertainty. Furthermore, the RCMs require an evaluation simulation to separate their biases from the biases induced by the GCMs in climate projections. Therefore, the regional climate models (RCMs) had to be evaluated against observations. This was achieved by downscaling the ERA-Interim [7] reanalysis data from 1989 to 2009. The European results (EURO-CORDEX, https://www.euro-cordex.net) were published e.g. by [34] and [14, 15] for the Weather Research and Forecasting (WRF) model [29] simulations of the University of Hohenheim at HLRS, and by [21, 32] and [20] for the whole EURO-CORDEX Ensemble. Afterwards, the GCM climate simulations are downscaled with the RCMs for the historical period until 2005 with observed greenhouse gas concentrations and the projections from 2006 to 2100 assuming the RCPs’greenhouse gas concentrations (e.g. [16]).

In order to densify the CORDEX ensemble and to increase the confidence in the estimation of extreme values such as hot days, the BMBF-funded (Bundesministerium für Bildung und Forschung) project ReKliEs-De (Regionale Klimaprojektionen Ensemble für Deutschland) was established. By the end of the REKliEs-De project in December 2017, 22 simulations for the RCP8.5 scenario and 6 additional simulations for the RCP2.6 were completed to complement the EURO-CORDEX ensemble. Five of these simulations were conducted by the University of Hohenheim with WRF at HLRS. This is the largest RCM projection ensemble until 2100 at the high resolution of 12 km, which has ever been produced for a specific region of this planet yet. With a total of 37 simulations of the RCP8.5 scenario and 14 simulations of the RCP2.6 scenarios, it is now possible to produce robust climate change information on high spatial resolution for Germany. Within the framework of ReKliEs-De, special climate indices were calculated and analyzed in detail with a focus on Germany and its main river catchment areas [12]; http://reklies.hlnug.de/).

For many applications, information about climate indices on the daily and monthly scales at 12 km is provided with a high level of confidence. However, some climate change signals are required at higher spatial and temporal resolution, e.g., the simulation of the diurnal cycle of precipitation suffers from the still not available convection permitting (CP) resolution, for instance, in case of high impact weather in summer. Previous work already demonstrated that further downscaling to a higher resolution on the CP scale (grid increments of  3 km or less) is not only beneficial for improved weather forecasting (e.g. [1]) but also for regional climate simulations [34]. At this resolution, the simulation of land-surface heterogeneities and orography is improved and the convection parameterization can be omitted. This view is supported by recent climate studies on the CP scale over the Alpine region [28] and over southern UK [18]. However, it is expected that the model uncertainties at these scales, where local interactions including land-atmosphere feedback play a more prominent role, will have even stronger impact on the final results than at the coarser RCMs applied e.g. in EURO-CORDEX. A multi-model ensemble has been set-up as a CORDEX flagship pilot study (FPS) to investigate present and future convective processes and related extremes over Europe and the Mediterranean [6]. We participate in this ensemble and the ReKliEs-project with WRF simulations in the frame of the WRFCLIM project at HLRS.

2 WRF Simulations at HLRS

For the regional climate simulations at 12 km resolution for Europe, the open source WRF model version 3.6.1 (http://www2.mmm.ucar.edu/wrf/users/) was applied with 5400 cores in hybrid mode (MPI+OpenMP) to achieve the best speed-up for our problem. The scalability of the WRF model is depending on the problem size (see [33] for more technical details). For the CP CORDEX-FPS test cases, WRF version 3.8.1 was applied with 240 cores to ensure all participating WRF configurations are applied in Europe with as little technical differences as possible. The CP climate evaluation simulation is now running with 1920 cores for central Europe including the southern half of Germany.

2.1 Climate Projections

Within the ReKliEs-De project future climate projections were carried out, applying four different GCMs with two different RCP scenarios as boundary forcing for the WRF model [24, 33]. The historical scenario of the GCMs covers the period from 1958 to 2005. The simulations of the historical period are forced by observed atmospheric composition changes of anthropogenic and natural sources. The projections carried out with the RCP scenarios covering the period from 2006 to 2100, represent mitigation scenarios that assume policy actions will be taken into account to achieve certain emission targets [30]. The numbers of the RCPs give a rough estimate of the range in the change of the radiative forcing due to projected GHG emissions by the year 2100 relative to the pre-industrial values. The RCP8.5 scenario is the so-called business-as-usual scenario which would lead to an approximated rise of the global temperature of around 4 \(^{\circ }\)C.

The RCP2.6 is the so called climate-protection scenario to keep the global temperature rise beneath 2 \(^{\circ }\)C. All regional climate models were applied to the EURO-CORDEX domain specified by CORDEX (see http://www.euro-cordex.net/). The analyses of ReKliEs-De is limited to Germany and its main river catchments (Fig. 1).

Fig. 1
figure 1

Model terrain height (m) in the model domain for  12 km horizontal resolution. The ReKliEs-De area is highlighted with a black contour line

Full size image

The WRF simulations for ReKliEs-De were started in 2016 at HLRS [33] and completed in October 2017. The computational resources from Juli 2017 to June 2018 are listed in Table 1. WRF was applied with the land surface model NOAH [3, 4], the Morrison two-moment microphysics scheme [25], the Yonsei University (YSU) planetary boundary layer (PBL) scheme [11], the Kain-Fritsch-Eta convection scheme [17] and the radiation parameterization scheme CAM for longwave and shortwave radiation [5].

Table 1 Computing resources of WRFCLIM from July 2017 to June 2018 on Cray XC40
Full size table

The raw model output led to an output size of 1.6 PB stored at HPSS and postprocessed to obtain the EURO-CORDEX format required to archive and publish the daily and monthly data on the archive system of ESGF (Earth System Grid Federation, https://esgf-data.dkrz.de/projects/esgf-dkrz/). The postprocessing and analyses of the diurnal cycles of variables like precipitation is ongoing and step by step the raw data will be deleted from the HPSS. In Sect. 3 highlights of this treasure of climate simulations are reported.

These activities required the development of respective scripts, processing steps, and infrastructures which are now available at HLRS. Therefore, we can state that we realized a worldwide known hub on earth system modeling and climate projections in southern Germany which is a well requested partner for upcoming national and international projects and contributes significantly to one of the most important problems of human kind in this century: this is climate change.

2.2 Convection Permitting Climate Simulations

Since WRF version 3.8.1 it is possible to run WRF also with the more sophisticated NOAHMP [27] land surface model with OpenMP and hybrid mode respectively. Based on our previous sensitivity studies with respect to the representation of land-atmosphere feedbacks and the PBL moisture profiles [22] at CP scale, for these simulations the MYNN PBL scheme [26] with NOAHMP was chosen. The RRTMG scheme was used for longwave and shortwave radiation [13], Thompson microphysics scheme [31], GRIMS (Global/Regional Integrated Modelling System) scheme for shallow convection, and Grell-Freitas ensemble scheme [9] for deep convection parametrization in the outer domain. Prior to CP climate model simulations within a CORDEX FPS for Europe, an experiment was designed in order to illustrate advantages and possible problems of this very high resolution RCMs in a multi-model ensemble [6]. For the experiment three case studies with strong precipitation events each with different background forcing, have been chosen to be conducted in so called two modes:

1. with weather like initialization (WL)

2. with the climate mode initialization (CM).

We performed the WRF simulations for CORDEX-FPS at HLRS. All the simulations were forced with ERA-Interim reanalysis data, and carried out over the EURO-CORDEX domain at 12 km, and further downscaled to 2.8 km covering central Europe (Fig. 2). WL simulations were initiated shortly prior to the event and carried out for 5–7 days depending on the case study. CM simulations were initiated with a longer time-frame prior to the event continuing for mostly \(\sim \)1 month.

Fig. 2
figure 2

Model terrain height (m) for a 12 km and b 2.8 km horizontal resolutions for the CORDEX-FPS convection permitting climate simulations

Full size image

Even though the model is able to run in hybrid mode (MPI+OpenMP), it was decided that all the ensemble members run with different WRF configurations by the participating groups run the model with 240 cores. Therefore this was also applied for these case studies at HLRS. Within the 24 hour wall-time \(\sim \)1 month was simulated. The simulations were completed in autumn 2017 and analyzed as part of the multi-model ensemble [6].

Since February 2018 such an evaluation run from 2000 to 2014 is ongoing, with 1 year of spin-up (1999). By the end of April 2018, 3 years are simulated. The model setting is similar to the setting used for the test cases, only the coarser domain is changed to a horizontal resolution of 15 km due to the results obtained from the test cases. Further the simulations are run at hybrid mode with 1920 cores. Table 2 summarizes the simulation details.

Table 2 Simulation details for the CORDEX-FPS convection permitting climate simulations
Full size table

3 Results

3.1 Climate Projections

In the business-as-usual scenario (RCP 8.5) the results reveal for Germany an increase of the annual mean temperature by 4 \(^{\circ }\)C (ensemble bandwidth of 2.8–5 \(^{\circ }\)C) at the end of this century in comparison with the end of the 20th century (Fig. 3). This implies a significant increase of heat days (maximum temperature above 30 \(^{\circ }\)C) and decrease of frost days (minimum temperature below 0 \(^{\circ }\)C) (Fig. 3). In Germany winter precipitation will increase and due to the warming this will be rather rain than snow. South western Germany will become drier in summer ([23], http://reklies.hlnug.de/).

Fig. 3
figure 3

Temporal development of the annual mean temperature change in the scenarios RCP8.5 (red) and RCP2.6 (blue) calculated for the ReKliEs-De domain from the ensemble members run for both scenarios. Source http://reklies.hlnug.de/

Full size image

In case of the climate protection scenario (RCP2.6) this increase would only be 1 \(^{\circ }\)C (bandwidth of 0.7–2.4 \(^{\circ }\)C) with a corresponding reduction of the number of extreme events. The strong difference in the evolution of mean temperature is demonstrated in Fig. 3. Obviously, the RCP8.5 must be considered as very dangerous for the society whereas the RCP2.6 scenario will lead to an increase of temperature which falls in the goals of the Paris Agreement in 2015. Therefore, the ReKliEs-De results do not only contain information for adaptation in different areas such as hydrology and agriculture but have also political implications for decision makers.

Fig. 4
figure 4

Number of ice days id (max. temperature > 0 \(^{\circ }\)C) for different single ensemble members of RCP8.5 scenario for the ReKliEs-De domain with 11 year running mean. Source http://reklies.hlnug.de/

Full size image
Fig. 5
figure 5

Change in summer (JJA) precipitation (%) between 1971\(\textendash \)2000 and 2071\(\textendash \)2100 in case of RCP2.6 (white) and RCP8.5 (gray) for the ReKliEs-De domain, Germany and the larger river catchments within the ReKliEs-De domain. Source http://reklies.hlnug.de/startseite

Full size image

Within ReKliEs-De, 24 climate indices were calculated and data and graphics were made publically available (see http://reklies.hlnug.de/). Two of these indices are displayed exemplary in Figs. 4 and 5. Ice days are defined as number of days where maximum temperature stays below 0 \(^{\circ }\)C. From 1960 to 2100, ice days decrease from around 30 to less than 10 ice days per year. Although, the single simulations members revealing a large spread from 20 to 40 days in the 1960s towards the end of the 21st century, all members project to have less than 10 ice days per year. Figure 5 shows the decrease in summer precipitation for the climate protection scenario RCP2.6 and the business-as-usual scenario RCP8.5. The decrease of precipitation in the RCP8.5 scenario in most catchments is larger than 15%.

The climate indices reveal details about expected climate change that will impact our society, agriculture, health and economy at 12 km horizontal resolution and due to the large ensemble of model simulations with a high confidence. The data is now publically available and used by impact researchers and public authorities to assess climate change impact. Further they urge the need to follow the climate protection scenario [19].

3.2 Convection Permitting Climate Simulations

Some climate change signals are required at higher spatial and temporal resolution, e.g., the simulation of the diurnal cycle of precipitation and extreme events require convection permitting (CP) resolution. The CP CORDEX-FPS test cases were analyzed by [6] for precipitation onset, intensity, and the location of maxima for different regions in comparison with available observations. First case (C1) referred to an Intense Observation Period 16 (IOP16) during HyMeX measurement campaign in September and November 2012. Heavy precipitation event was observed from 23 to 28 October 2012 in southern France and NW Italy. Case two (C2) is a convective orographic precipitation event in Austria with weak but persistent large-scale forcing. Strong precipitation events were observed from 22 to 25 June 2009. Case 3 (C3) is the Foehn case observed in the period between 3 and 7 November 2014, when the slow eastward evolution of the trough caused persistent precipitation over the Alps with daily precipitation locally exceeding several hundreds of mm and reaching maximum of around 500 mm.

Fig. 6
figure 6

Total accumulated precipitation [in mm] during the event C1 (a), C2 (b), and C3(c). Observations are given in the top row (1), the WL multi-model ensemble mean in the second row (2), and the CM multi-model ensemble mean (3). From [6]

Full size image
Fig. 7
figure 7

Time series of accumulated precipitation (in mm on the y-axis) during the C2 event: a time series of the accumulated precipitation for each model and observations, b Time series of WL (blue) and CM (red) multi-model ensemble means, in comparison with observations (black) over the area of interest. From [6]

Full size image

The ensemble comprised 18–21 members, depending on the test case. For the C1 case both WL and CM ensemble means underestimate the observed accumulated precipitation for the indicated period (Fig. 6a1–a3). The underestimation is more pronounced for the CM simulations, but the location of the maxima is well represented with both ensemble means. In C3 case, both the location of the precipitation maxima and the total accumulated precipitation amount is well represented with WL and CM ensemble means (Fig. 6c1, b2, b3). The most interesting case is the case C2. The CM ensemble mean strongly underestimates the observed precipitation (Figs. 6b3, 7a, b). The spread of the both ensembles is large, and most of the CM simulations completely missed the event with accumulated precipitation close to 0 mm (Fig. 7a). The difference in behavior between the two ensembles is evident in Fig. 7b, where the CM ensemble mean shows an underestimation around the 60% of the correspondent observed curve. A detailed investigation of these test cases is still undergoing, but few factors appear to be responsible for the results of the most interesting C2 case: (1) problematic can be that the event is too close to the domain boundaries, and the involved modelling groups are using varying sponge layer depths and nesting strategies; (2) a relatively weak background synoptic state of the event would decrease the large scale forcing compared to local ones, and thus increase diversity across models [6].

4 Conclusion

The RCM simulations within ReKliEs-De revealed the necessity of a large number of ensemble members at high resolution for Europe to assess climate change at the regional scale. Considerably more confidence and robustness in the expected climate change signals were achieved by the evaluation of the large ensemble. Also, the required ensemble member number to receive robust information was derived in ReKliEs-De. Some climate change signals like mean temperature change can be assessed with a limited number of ensemble members, but some climate indices like tropical nights will require even more than the studied 37 model simulations to receive robust signals. The simulations from 1958 to 2100 with a model time step of 60 s and a 3 hourly model output frequency with WRF were facilitated at the CRAY XE6 and XC40 from summer 2016 to autumn 2017 (e.g. [2]. Each simulation needed 3600 hours walltime on 5400 cores. The resulting 400 TB raw output required postprocessing, archiving and publication on the ESGF archive system. This is still ongoing. The ReKliEs-De project demonstrated the importance of the HLRS infrastructure to enable the required five WRF climate projections within the project timeframe. This was only achieved by running the simulations in parallel, i.e. 27,000 core hours per job.

The results from ReKliEs-De highlight the importance of assessing climate change signals with a multi-model ensemble, WRF being one of them. The results of ReKliEs-De which are also published on http://reklies.hlnug.de/startseite, indicate a general increase of weather extremes in Germany. This fact is of great importance especially in agriculture, since large problems will evolve in the cultivation of different field plants. The development and timing of the sowing and harvesting of such plants are strongly dependent on optimum temperature ranges. The preliminary results from CORDEX-FPS test cases imply that for the long-term simulations which are at the moment ongoing, we can expect varying ranges of responses depending on types of the convective events, as well as to the specification of the model deployed [6]. In CP-RCM it has been seen that the precipitation pattern are moving with the local upper level flow, which is not the case in coarser RCM simulations where the precipitation remains locked at the mountain tops [10]. This has been also seen in these test cases. Such moving precipitation is a more realistic representation, which can be related primary to the convection that is resolved in CP-RCM, and not parameterised as is the case in coarser resolution models. Also such models have more detailed representation of land surface heterogeneity, which may strongly affect representation of precipitation distribution through land-atmosphere feedback processes. Further steps will include more thorough analysis of the impact of such processes on cloud formation and precipitation using these the CP-RCM simulations and various land-atmosphere feedback metrics. The currently running ensemble of CP climate simulations of the CORDEX-FPS will be evaluated at the end of this year before this CORDEX-FPS starts in its second phase of CP climate projections in 2019.