1 Introduction

The science community agrees unequivocally that climate is warming. Since the 1950s, the observed changes are unprecedented (IPCC 2014). Moreover, continued emission of greenhouse gases (GHGs) will cause further warming and long-lasting changes in all components of the climate system. The anticipated change in climatic parameters is expected to alter the hydrologic cycle and the frequency and magnitude of hydrological extremes in various regions (Christensen and Lettenmaier 2007; Sheffield and Wood 2008; Gosling et al. 2011; Hagemann et al. 2013; Aich et al. 2014; Giuntoli et al. 2015). Despite the advancement of state-of-the-art global circulation models (GCMs) and hydrological impact models, our current understanding of future climate and its impacts on water resources is beset with the inherent uncertainties in emission scenarios, GCM structures and initial conditions, downscaling methods, hydrological model (HM) structures, and their parameters.

GCMs typically represent the atmosphere, ocean, land surface, cryosphere, and biogeochemical processes and solve the equations governing their evolution on a geographical grid covering the globe. Some large-scale processes are represented explicitly within GCMs while processes taking place on scales smaller than the typical grid size of a GCM are represented by simplified parameterizations. Thus, various GCMs use different representations of the climate system depending on the modeling institute. This leads to “climate model structural uncertainty” (Gosling et al. 2011), meaning that climate projections for a single greenhouse gas emission scenario differ for different GCMs.

In addition to GCM structure and emission scenarios uncertainties, differences between HM structures also represent an important source of uncertainty. Previous studies that applied global HMs (Giuntoli et al. 2015; Hagemann et al. 2013) and regional HMs (Vetter et al. 2015; Jiang et al. 2007) suggested that multiple HMs should be used for climate change impact studies. Gosling et al. (2011) found up to 25% difference in mean annual discharge depending on HM selection. They also observed that the spread arising from the selection of GCMs is larger than that arising from the selection of HMs. However, a study by Hagemann et al. (2013) showed that for AET (in many regions) and for discharge (in some regions of the globe) the spread resulting from the choice of HM is larger than the spread originating from the climate models. Furthermore, Poulin et al. (2011) noted more significant uncertainty due to HM structure than HM parameters.

Quantifying the different sources of uncertainty is very important for decision-making. Recommendations based on single model results may be highly uncertain whereas application of multiple models will highlight the different sources of uncertainties. In case of high uncertainty, decision-makers should rather opt for “no regret” measures, such as rehabilitation of natural flooding areas or prevention measures. In case multiple models project similar changes, more robust decisions can be taken.

Several studies have researched the potential impacts of climate variability and/or change on water resources of the Upper Blue Nile (UBN) basin (Conway and Hulme 1993; Conway 1996; Elshamy et al. 2009; Beyene et al. 2009; Taye et al. 2011; Mengistu and Sorteberg 2012; Aich et al. 2014), among others. There is a general consensus among the GCM projections that temperature will increase in the UBN (Elshamy et al. 2009, 2012; Beyene et al. 2009; Taye et al. 2011). However, future changes in precipitation are uncertain and depend on the GCM used. Since discharge is very sensitive to changes in precipitation in the UBN basin (Elshamy et al. 2009), the uncertainty of the projected discharge will likely complicate the long-term water resources planning and management in the region.

Most of the methods used in previous studies over the UBN for assessing the magnitude of climate change impacts were based on running one HM driven by various GCMs as input data. However, Taye et al. (2011) used two conceptual models to analyze climate change impacts in the headwater catchments of the Nile basin and found that HM performance influences the future impact projections. Yet, these two models do not sufficiently represent state-of-the-art model structure and level of complexity. Therefore, we argue that an ensemble of HMs should be used for a more credible assessment of the projected changes in discharge and actual evapotranspiration (AET) during impact evaluation studies.

In this respect, we investigated the likely impacts of future climate change on water resources of the UBN basin using an ensemble of six HMs: Soil and Water Assessment Tool (SWAT) (Arnold et al. 1998), Soil and Water Integrated Model (SWIM) (Krysanova et al. 1998), Water–Global Assessment and Prognosis (WaterGAP3) (Alcamo et al. 2003), mesoscale hydrologic model (mHM) (Samaniego et al. 2010; Kumar et al. 2013), variable infiltration capacity (VIC) (Liang 1994), and Hydrologiska Byråns Vattenbalansavdelning (HBV) (Lindström et al. 1997). Additionally, we evaluated three major sources of uncertainty due to GCM structure, representative concentration pathway (RCP) scenarios and HM structure on the projected discharge, and AET. This study contributes to the Inter-Sectoral Impact Model Intercomparison Project (ISI-MIP) initiative (phase 2), which is a community driven modeling effort to assess global and regional impacts of climate change.

2 Study area

The Nile River, the longest river in the globe, is a crucial water resource for agriculture, domestic use, and power generation for the Nile Basin countries. The White Nile, which originates from Lake Victoria, and the Blue Nile, which starts at Lake Tana, join at Khartoum to form the Nile River. It then continues flowing towards the Mediterranean Sea. The Blue Nile contributes around 80% of the total discharge during the high discharge season (June–September) and 50% of the total annual discharge of the Nile (Nawaz et al. 2010), although its area is only 10% of the Nile drainage area (Yates and Strzepek 1998).

This study focuses on the UBN basin until the El Deim gauge station that covers an estimated area of 176,000 km2 located in Ethiopia. This basin is characterized by undulating topography with an elevation that varies from approximately 4000 m to 700 m a.s.l. The rainfall distribution in the UBN basin is mono-modal (Figure 1S in the Supplementary material) with average annual rainfall of approximately 1300 mm/year (1951–2001) whereas more than 90% of rain is falling from June to September (Berhane et al. 2013). The annual average temperature in the basin is about 21 °C. The average monthly discharge at the El Diem station varies between 200 and 5500 m3/s with an annual average of 3600 m3/s (Nawaz et al. 2010).

3 Data and models

3.1 Hydrological models and meteorological data

The UBN basin was modeled using six HMs: SWAT, SWIM, WaterGAP3, mHM, VIC, and HBV independently by each participating research group. For a short description of each hydrological model and their input spatial datasets—elevation, land cover, and soil map—please refer to Tables 1 and 2 in the editorial introductory paper of Krysanova and Hattermann (this SI). WaterGAP3 is a regional version of the global model WaterGAP, developed with an objective of improving efficiency in simulating water cycle, where two parameters were adjusted during calibration.

Table 1 Data from global and regional datasets used for calibration and validation of hydrological models in most case studies
Full size table
Table 2 A short description of nine models applied for the regional-scale impact assessment and intercomparison in 12 river basins
Full size table

The HMs were forced with the reanalysis climate forcing data from WATCH (Weedon et al. 2011) for the time period 1971–2001. The WATCH dataset has a resolution of 0.5° × 0.5° and includes precipitation, maximum and minimum temperature, wind speed, radiation, and relative humidity at a daily time step. The calibration and validation of the HMs were carried out using Global Runoff Data Center (GRDC) river discharge data for the El Diem station located at the Ethio-Sudan border.

3.2 Climate projections

To understand the complexity of the interaction between anthropogenic activities and the climate system, scientists have developed scenarios which help to evaluate the uncertainty related to human activity contribution to climate change (Moss et al. 2010). In our analysis, the third-generation IPCC scenarios—RCP2.6, RCP4.5, RCP6.0, and RCP8.5—were used, which were adopted by the Intergovernmental Panel on Climate Change (IPCC) for its fifth Assessment Report (IPCC 2014). The RCPs are defined as pathways which provide time-dependent projections of GHG concentrations until 2100. In the RCP inception phase emphasis was given not only to the long-term GHG concentration but also to the path taken to reach the outcomes. The RCPs are representative meaning they are few of the several possible scenarios that have similar outcomes in terms of radiative forcing and emissions characteristics.

The GCMs were used to project climate under these scenarios. This study used climate projections from four Coupled Model Inter-comparison Project Phase 5 (CMIP5) GCMs: GFDL-ESM2M, HADGEM2-ES, MIROC-ESM-CHEM, and NorESM1-M. Unlike the previous studies (Elshamy et al. 2009, Nawaz et al., 2010 and Taye et al., 2011) for the UBN basin which applied statistical downscaling methods to downscale the GCM outputs, the GCM climate outputs in this study were bi-linearly interpolated in space to a 0.5° × 0.5° grid, and bias corrected (Hempel et al. 2013). The bias correction preserved the absolute changes in monthly temperature, and relative changes in monthly values of precipitation and other variables. For detailed information on the bias correction method, the reader is referred to Hempel et al. (2013).

4 Methods

4.1 Hydrological model performance evaluation

The goal of this paper is to compare and analyze differences in simulated discharge and AET using an ensemble of six HMs, four GCMs, and four RCP scenarios. The HMs were calibrated and validated by each participating research group with daily discharge data from the El Diem gauge station for a duration of 5–10 years in the period 1971–1980. For that the HMs were driven by WATCH climate dataset. We used statistical performance indices such as Nash-Stutcliffe efficiency (NSE) and percent of bias (PBIAS). In this study, the performance of the bias corrected GCMs in replicating the climate of the historical period was checked by comparing simulated discharge by each HM (forced by each GCM projection) after calibration and validation with the observed discharge at the El Diem station. We used relative deviation in mean (Δμ), relative difference in standard deviation (Δσ) and Pearson’s correlation coefficient (r) to measure the skill of the HMs in simulating the seasonal dynamics (Gudmundsson et al. 2012). Since evaluating the impacts of climate change on AET is one of the objectives of this paper, the closure of the water balance within HMs for the historical period was also assessed (based on the equation P ∼ Q + AET).

4.2 Climate change impact analysis

The mean annual and mean monthly hydrological fluxes (discharge from six HMs and AET from four HMs as data of only four HMs were available) and climate parameters (precipitation and temperature) in the mid future (2036–2065) and far future (2070–2099) were compared against the baseline period (1981–2010). This method allowed the detection of change signals. The significance of the calculated signals was evaluated using an unpaired statistical test with p = 0.05. The p values were calculated using scipy.stats.ttest_ind library within Python 2.7 (programming language). In the statistical test, two sample groups—for baseline and future periods—were created by combining all monthly or annual values of the examined variables. For example, to quantify changes in the projected mean annual discharge simulated by a HM for the far future period, we created two samples: one for the baseline and one for the far future period, each consisting of 480 annual mean discharge values (4GCMs × 4RCP scenarios × 30 years). We first calculated the change in the sample means and then tested if the change between the sample means was significant (with 95% confidence interval). This was repeated for each HM. The same approached was used to calculate change in mean monthly discharge (AET) grouped by GCMs and RCP scenarios.

4.3 Uncertainty analysis

Uncertainties in projected change in mean annual discharge (and AET) due to HM structure, GCM structure and RCP scenarios were quantified. To calculate the uncertainties, the projected changes in mean annual discharge (AET) calculated as described in Section 4.2 were compared within each group of HMs, GCMs, and RCPs to calculate the ranges (differences between the maximum and minimum values) in the group. These ranges for every group were reported as uncertainty in discharge (AET) related to HMs, GCMs, and RCPs.

5 Validation of hydrological models and climate projections

The comparison of the simulated discharge against the measured discharge at the El Diem gauge station with the monthly time step showed good or satisfactory performance for most of the HMs during the calibration and validation period using WATCH dataset. Detailed information on the performance of the hydrological models evaluated with 12 statistical indices can be found in Huang et al. (2016). For five of the calibrated models the NSE ranged from 0.75 to 0.84, and the PBIAS ranged from −1 to 7%. WaterGAP3 showed NSE of 0.27 and PBIAS of 70%, which could be explained by the fact that it was calibrated using only two parameters. Furthermore, as depicted in Figure 2S the HMs performed very well in the closure of the water balance. The small discrepancy in water balance closure in SWIM and SWAT models is partly attributed to water loss to the deeper aquifer. In summary, we can conclude that all models are in general able to simulate seasonal dynamics and annual water balance.

The evaluation of GCMs in replicating the historical climate was done by comparing simulated monthly mean discharge in the historical period driven by GCMs with the observed discharge; the results are depicted in Figure 3S in the online resources. Most of HM-GCM combinations reproduced the shape and timing of the mean annual discharge cycle well as indicated by the large r (Table 1S). However HBV and WaterGAP3 driven by GCMs overestimated mean annual discharge as indicated by high Δμ.

6 Results and discussion

6.1 Precipitation and temperature projections

Downscaling and bias correction of climate projections was done in the first phase of the ISI-MIP (Hempel et al. 2013) as mentioned in Section 3.2. This study used a set of 16 climate projections for the analysis; a combination of 4 GCMs (initially we considered a 5th GCM: IPSL-CM5A-LR but it was found to be an outlier and was thus excluded from our analysis) and 4 RCP scenarios. As precipitation and temperature are the main inputs for hydrological models, future projected precipitation was compared with baseline precipitation. As expected and in agreement with the findings of previous studies (Beyene et al. 2009), the change signal in temperature showed an increasing trend among all the GCMs. Figure 1 depicts the change in ensemble mean monthly precipitation (in mm) and the change in mean monthly temperature relative to the baseline period.

Fig. 1
figure 1

Left: seasonal change in ensemble (4 GCMs) mean precipitation relative to the baseline period. Right: change in mean monthly temperature relative to the baseline period. The red shade indicates 95% confidence interval for the change in ensemble means. The gray shade indicates the wet season in the Upper Blue Nile basin

Full size image

Regarding the mean annual precipitation in the future periods compared to the baseline, for most GCM-RCP combinations an increased precipitation was projected. We tested the significance of these projected changes using an unpaired statistical test with a significance level of p = 0.05. For the far future period, two of the four GCMs (HADGEM and NORESM) indicated a statistically significant increase in precipitation whereas for the other two GCMs the projected changes were not significant (Figure 4S, in the online resources). By the end of the century the projected changes in mean annual precipitation relative to the baseline period vary between 4.1% (HADGEM2) and 9.6% (NORESM). Overall, an increase in annual mean precipitation of up to 45 mm (16 mm) was projected for the far (mid) future period.

The seasonal analysis of projected precipitation showed a significant increase in ensemble mean precipitation at the end of the raining season (Fig. 1). The ensemble mean of projected changes in precipitation was about 2.8 and 4.65% for the mid and far future periods, respectively. Whereas the ensemble mean change in temperature averaged over all RCPs was about 1.7 °C for mid future and 2.6 °C for the far future period. All the GCMs projected an increase in annual mean temperature for both future periods. For the UBN, the wet season spans from June to September. In our seasonal analysis of projected precipitation a shift in the rainy season with a likely decrease of precipitation in July and August and a likely increase in precipitation in October was noted. As shown in Fig. 1, the change in the ensemble mean of projected precipitation was especially high at the end of the rainy season (indicated by the vertical grey shade).

6.2 Impact on mean annual discharge and evapotranspiration

Beside GCMs and RCP scenarios, the variation in magnitude of impacts of climate change on mean annual discharge also depends on the selection of the HM (Fig. 2). Figure 2 presents the percentage of changes in mean annual discharge (six HMs) and AET (four HMs) based on forcing from four GCMs and four RCP scenarios. The projected changes in mean annual discharge grouped by GCMs disagree in magnitude and direction for both future periods, with values ranging from −11.2% (MIROC_ESM_CHEM) to 4.3% (HADGEM2-ES) for the mid future and from −11.1% (MIROC-ESM-CHEM) to 12.4% (NORESM1) for the far future period. The HMs project changes in mean annual discharge, which are mostly not statistically significant. For the mid future period, all the HMs except SWIM projected a decrease in mean annual discharge whereas SWIM simulated an increase. But none of these changes were statistically significant (p = 0.05). For the far future period, HBV, mHM, and SWAT models simulated a decrease whereas SWIM, VIC, and WaterGAP3 simulated an increase in mean annual discharge. For this period, only the projection by SWIM (6.6 ± 3.5%) was found to be statistically significant (p = 0.05). The ensemble mean of the projected changes from all the HMs was about −1.8 and 1% for the mid and far future periods, respectively. In summary the projected increase in mean annual precipitation did not generate a significant change in mean annual discharge in most cases, and this can be attributed to the projected increase in mean annual AET. The trend in projected change in mean annual discharge from RCP2.6 to RCP8.5 was found to be the reverse of the trend for projected mean annual temperature (Fig. 1) as depicted in Fig. 2.

Fig. 2
figure 2

Percentage change in mean annual actual evapotranspiration (AET) (top) and mean annual discharge (bottom) grouped by global circulation model (GCM) (first column), representative concentrations pathway (RCP) scenarios (second column), hydrological models (HM) (third column), and total ensemble mean (fourth column) for the 2036–2065 period and the 2070–2099 period relative to the baseline period. Points indicate means and error bars indicate 95% confidence interval for the percentage of change in means. GFDL GFDL-ESM2M, HADGEM HadGEM2-ES, NORESM NorESM1-M, MIROC MIROC-ESM-CHEM

Full size image

AET simulation by HMs depends on the selection of the potential evapotranspiration (PET) estimation method as well as model structure which affects the soil moisture accounting (Alemayehu et al. 2015). In the HMs used in this study, a total of six different PET methods were applied (Table 2S of online resource). Figure 2 shows the statistically significant (at p = 0.05) changes in mean annual AET in both future periods compared to the baseline. This is regardless of the choice of GCMs, RCPs scenarios, HMs, and PET method. All the HMs simulated an increase in mean annual AET ranging from 1.2 ± 0.8% (SWAT) to 3.6 ± 1.8% (SWIM) for the mid future, whereas for the far future the ranges are higher: from 1.9 ± 0.8% (SWAT) to 4.4 ± 1.3% (WaterGAP3), though we noted that results from the three HMs were very close (Fig. 2). These results are logical given the projected increase in temperature and precipitation (the dominant controlling factors) in the UBN basin. The ensemble mean for the projected changes in AET were 2.3 and 3.6% for the mid and far future periods, respectively. A comparison between the magnitudes of change in projected annual mean precipitation and AET indicated that there was more water available, on average +1 mm/year for the mid future and +16 mm/year for the far future periods.

6.3 Impacts on seasonal discharge and evapotranspiration

Figure 3 presents simulated changes in mean monthly discharge relative to the baseline (1981–2010), together with the 95% confidence interval, separately for every HM. Unlike the difference in direction of projected changes in mean annual discharge by the HMs, the projected change in mean monthly discharge by all HMs except SWAT indicated a reduction in wet season (until September) discharge. Furthermore, for the months from October to November statistically significant changes (p = 0.05) in mean monthly discharge in both periods were projected by all HMs. The reduction in the projected wet season precipitation (see Fig. 1) and the associated decline in wet season discharge in the UBN basin indicate the prevailing influence of precipitation on discharge in the basin. Similar observation was noted by (Conway 1996) with three GCM experimental projections and one HM. The projected relative changes in mean monthly discharge during the dry season were found to have higher uncertainty compared to the wet season (which is mostly related to the GCM input); this is depicted by the width of the 95% confidence interval as shown in Fig. 3.

Fig. 3
figure 3

Percentage changes in mean monthly discharge for the period 2036–2065 (left-top) and 2070–2099 (left-bottom) relative to the baseline as simulated using HMs. Ensemble mean of % change in average monthly discharge simulated by HMs (right). For each impact model simulation from 4 GCMs and 4 RCPs scenarios (in total 16) were accounted. In the left panel, the solid line denotes the ensemble mean of 16 simulations for each model. In the right panel, the solid line denotes the ensemble mean of 96 simulations. The red-shaded area represents the 95% confidence interval

Full size image

Figure 4 shows the projected mean monthly changes in AET as simulated by multiple HMs for the mid future and far future periods. Each simulation is an ensemble driven by four GCMs for four RCP scenarios. It is apparent from all HM simulations that the projected changes in AET follow similar patterns as the projected changes in precipitation, for both mid future and far future periods. However, the increase is likely to be more pronounced in the last period.

Fig. 4
figure 4

Changes in mean monthly actual evapotranspiration (AET) in mm as simulated by multiple hydrological models for the period 2036–2065 and 2070–2099 (left-top and left-bottom) relative to the baseline. Ensemble mean of % change in average monthly AET simulated by HMs (right). The red-shaded area represents the 95% confidence interval

Full size image

It is worth noting that the HMs used in this study do not always agree on the direction of mean monthly AET changes in the UBN basin. In addition, the disagreement on the projected mean monthly AET could be partly attributed to the HMs structure. In a previous study for this basin, Elshamy et al. (2009) found an increase in wet season AET for the 2081–2098 period using 17 GCMs and one impact model for the A1B scenario, which agrees with the mHM results in our study. In our study, mHM results show an increase in AET over the wet season, and the SWAT and WaterGAP3 models suggest an increase in the second half of the wet season, which contrasts with the SWIM model projections. SWAT and mHM used the Hargreaves methods for PET estimation. As can be seen in Fig. 4, the AET projections simulated by these two HMs have similar patterns whereas projections from two other HMs differ. Previous studies noted the influence of the PET estimation method on the projected discharge and AET (Thompson et al. 2014) and suggested that simplified models could be better for the PET estimation (Kay and Davies 2008; Alemayehu et al. 2015) than the complex and data intensive methods.

6.4 Evaluation of the major sources of uncertainties

Figure 5 gives an overview of the uncertainties in discharge and AET by plotting the ranges in projected means for each uncertainty source (GCMs, HMs, and RCP scenarios). These ranges are calculated as explained in Section 4.3. In our analysis, as also noted in several studies (Chen et al. 2011; Exbrayat et al. 2014; Gosling et al. 2011) among others, the dominant source of uncertainty in the simulated mean annual discharge stems from the GCM structure. If the fifth GCM (the outlier) would be included, the uncertainty related to GCMs would be even higher as noted by Eisner et al. and Vetter et al. (this SI). The contribution of uncertainty due to HM structure on the simulated mean annual discharge is higher for both future periods than the uncertainty due to RCP scenarios.

Fig. 5
figure 5

The range of uncertainty in the projected percent of change in mean annual discharge and AET for period 2036–2065 and 2070–2099 relative to the baseline. Note that the higher the height of the bars the larger the uncertainty

Full size image

For the simulated mean annual AET the contribution of uncertainty due to GCM structure, HM structure and RCP scenarios were found to be more comparable. The projected mean annual discharge changes spanned from 12.4% (NORESM) to −11.2% (MIROC) for the far future compared to the baseline period (results shown in the bottom first panel of Fig. 2). The HMs show mostly statistically insignificant changes in mean annual discharge by the end of the twenty-first century ranging from −2.7% (mHM) to 6.6% (SWIM). Given the potential uncertainty in HM structure, the use of a single impact model in the evaluation of climate change impacts would likely lead to biased conclusions (as noted previously in this study). Therefore, future climate change impact studies should consider multiple HMs. In general, the projected changes in mean annual AET reveal lower uncertainty compared to the projected changes in mean annual discharge (Figs. 2 and 5). The mean monthly AET however is strongly influenced by uncertainties related to the HMs.

We recognize that our uncertainty analysis does not encompass the entire cascade of uncertainty sources found during investigations of climate change impacts on hydrology. The uncertainties stemming from downscaling methods, GCM initial conditions as well as HM parameters were not addressed in this paper. However, the three major contributing sources of uncertainty in projections of the mean annual discharge and AET were analyzed. Figure 5 summarizes the ranges of projected percentage changes in mean annual discharge and AET grouped by GCMs, RCPs, and HMs.

7 Summary and conclusions

The main objective of this study was to investigate the likely impacts of climate change on the major water balance components in the Upper Blue Nile basin. In addition, the uncertainties related to HMs structure, GCMs structure, and RCP scenarios selections on simulated mean annual discharge and actual evapotranspiration were analyzed. Based on our analysis of climate projections for the UBN basin, we found that two out of four GCMs showed a statistically significant increase in annual precipitation at the end of the century, and the wet season (June to September) will likely to be extended until the end of November. An increase in annual mean precipitation of up to 45 mm (16 mm) was projected for the far (mid) future period. All the GCMs projected an increase in annual mean temperature for both future periods.

Most of the HMs showed statistically insignificant changes in simulated mean annual discharge. Only SWIM model projected a statistically significant increase in mean annual discharges of up to 6.6% for the far future period (2070–2099). However, from the seasonal analysis of discharge, we noted that significant discharge changes may occur by the end of the wet season. All the HMs generally simulated an increase in annual mean AET in the mid future and far future periods. An increase in annual mean of 3.6% was projected for the far future period based on ensemble mean simulations of HMs.

The differences in GCM structure were the dominant source of uncertainty in discharge projections in the UBN basin, particularly by the end of the century. If the fifth GCM (the outlier) would be included, the uncertainty related to GCMs would be even more dominant. The uncertainty due to HMs was also found to be quite significant and higher than that arising from the RCP scenario selection. In summary, this research emphasized that HMs contribute notably to uncertainty in projections of discharge and AET in the UBN basin; therefore, future climate impact studies should consider using an ensemble of HMs rather than making assessment based on a single hydrological model that can lead to biased conclusions. In multiple model applications, we suggest to take into account differences in model performance. For that a weight factor (based on the model performance in historical period) as applied by Elshamy et al. (2012) could be used.