An important component of a multi-model dataset is the possibility to characterize the multi-model agreement or consistency. While such characteristics do not imply quality or skill, it can provide an overview of the regions and variables where datasets strongly disagree. This information can be used either by the modelling community to focus on particular aspects of their models or by users as a first order uncertainty estimate of the multi-model ensemble. The agreement metric selected here is the SNR, which compares the signal to noise levels by relating the ensemble's variance to that of the individual members, which has been widely used as a classical measure of predictability in seasonal forecasting . SNRs were calculated for three model variables: evapotranspiration, runoff and root zone soil moisture plus an ensemble of global precipitation datasets. See Appendix A for a detailed description of the SNR calculations.

We performed the calculations with monthly mean anomalies to focus on the model agreement in terms of intra-seasonal to inter-annual variability. Since all models were driven by the same atmospherical conditions, low values of SNR can be directly associated with differences in the representation of processes such as energy partitioning and runoff generation – i.e. ensemble uncertainty. However, since one single forcing was used, the ensemble is missing an important source of uncertainty: the driving data. Precipitation is likely the main source of uncertainty; it is very important for the terrestrial water balance while at the same time it is difficult to observe both locally and remotely. To put our results into perspective, we also computed the SNR of an ensemble of precipitation datasets including three atmospheric reanalysis datasets (ERA-Interim: , JRA55: , MERRA: ), two datasets based on rain gauges (GPCC: , CRU TS3.10: ) and two datasets derived from remote sensing data (GPCP: , CMAP: ).

The SNRs were computed for the period January 1980 to December 2012 by removing the mean annual cycle in each grid point from each ensemble member such that y‾ and y‾i are both equal to zero; see Appendix A. Due to the differences in the representation of soil moisture storage among the models, the ensemble mean is dominated by those models with a higher water holding capacity and hence larger absolute soil moisture variability. Therefore, the root zone soil moisture was first transformed to percentiles before performing the calculations for each model . The SNR varies between 0 and +∞ with values below 1 indicating that the inter-model variability is larger than the ensemble mean variability, i.e. low inter-model agreement, and values above 1 indicating a high inter-model agreement.

The multi-model consistencies in terms of inter-annual variability evaluated by the SNR are shown in Fig. a–c for the different variables, and aggregated by climate types in Fig. a–c. The results highlight regions with a low inter-model agreement over (i) snow-dominated regions (runoff, evapotranspiration and root zone soil moisture) and (ii) tropical rainforest and monsoon regions (evapotranspiration), whereas the temperate areas show a high inter-model agreement. Comparing these results with the precipitation datasets' agreement (Figs. d and d), which were not included in the driving data, the large uncertainty in the tropical areas is not reflected in the runoff or root zone soil moisture. On the other hand, there is little disagreement in the precipitation datasets in cold regions, which could be caused by the fact that in these regions we rely mostly on reanalysis data sources, while the multi-model ensemble contains a large spread. The SNRs suggest that over cold regions the multi-model ensemble is generating a large spread (likely due to the different treatment of cold processes among the models) while over the tropical areas some of the multi-model agreement might be underestimating the actual uncertainty by neglecting the driving data uncertainty in the ensemble generation.

We use the ILAMB system (The International Land Model Benchmarking Project; ) to compare the model results against benchmark data mostly derived from satellite remote sensing, of evapotranspiration (ET), terrestrial water storage anomaly (TWSA), soil moisture anomaly (SMA), snow water equivalent (SWE) and snow cover fraction (SC). For evapotranspiration both the GLEAM-V2a and V3b datasets and the MODIS evapotranspiration estimates were used while for soil moisture anomaly we used the combined active+passive microwave ESA CCI soil moisture dataset . The snow cover dataset was obtained from the Interactive Multisensor snow and ice mapping System (IMS; ) while for SWE we used GLOBSNOW-2 . The TWSA dataset was obtained from GRACE data (JPL; ). The complete benchmark results are available at the dataset storage entry page at https://github.com/earth2observe/water-resource-reanalysis-v1 while a summary is given in Appendix B, Table  to . A project report presenting in-depth verification is provided in the Supplement.

ILAMB provides a scoring system to relate modelled results to reference datasets. In the ILAMB system multiple performance metrics are calculated, and additionally these metrics are converted to scores ranging between 0 and 1 to facilitate comparison and averaging. In this study three performance metrics are calculated for each of the five model variables evaluated (ET, TWSA, SMA, SWE, SC): total bias, root mean square error (RMSE) and phase difference (difference in months between peak values); furthermore a total of five 0–1 scores are calculated, for global bias, RMSE, seasonal cycle, spatial distribution and inter-annual variability, plus a 0–1 overall score that summarizes them. The metrics and scoring are explained in detail in the ILAMB documentation (http://earth2observe.github.io/water-resource-reanalysis-v1/assets/pdf/ILAMB_metrics_document.pdf). For the case of anomaly variables (TWSA and SMA), the bias and spatial distribution scores are calculated over the standard deviations of the variables, and not directly over the anomalies. The current study is not an exhaustive test of the performance of the different models, instead we focus on the multi-model ensemble, commenting on specific models only when this is thought to be important to the ensemble as a whole. Apart from the global level, the ILAMB results are also available for a number of predefined regions (biomes; see Fig. ).

Although there are a number of uncertainties associated with TWSA as estimated by GRACE measurements , resulting from the uncertainty of the GRACE data itself and the leakage corrections, the results provide an independent means of evaluating our model results. Both the peak month phase difference and the ensemble mean bias are negative for most models compared to GRACE (see Fig. ) which could indicate that these models lack a store such as rivers or lakes or that the size of the included stores is inadequate. Table  lists the models and the variables used to calculate TWSA in this study. Results for the WaterGAP3 model should be interpreted with caution because it lacks total moisture which is a large store in many models. RMSE is largest in high-precipitation regions and lowest in dry areas (Fig. , bottom panel). Overall, total scores for TWSA are close between the models, while the highest values are recorded for the ensemble mean, suggesting that the lowest-scoring models do not have a large detrimental effect on the ensemble mean. Although the scores for the TWSA (ranging between 0.49 for SWBM and 0.60 for HTESSEL) are similar at the global level, there are larger differences between the models at the regional level. The negative phase difference for nearly all models indicates that the peak in the anomaly in the models occurs earlier than in the GRACE data. This could point to a general underestimation of the TWSA (for example the groundwater component) resulting in a system that reacts too quickly. In addition, the negative phase difference is strongest in the cold regions indicating that snow modelling might be an important factor here. There are large regional differences in the phase difference performance showing best results in Africa (below the Sahel), Australia, India, and South America north of 25∘ S. There is no single model that performs best in all regions.

For evapotranspiration our results compare better to the GLEAM products (mean model total score of 0.83 for both products) but less so to the MODIS product (mean model score of 0.78; see Table ). Most models (and the ensemble mean) evaporate more water than the remote sensing based estimates. As shown by it is still difficult to determine the quality of global ET products. The driving force behind the ET estimates by the multi-model ensemble is provided by the WFDEI (based on ERA-Interim) which is shown to have relatively high ET . It is beyond the scope of the current study to examine the differences in ET between the models, but the choice of calculation method of potential ET within the models may already account for a large spread (see e.g. , who used the WFDEI forcing to calculate FAO Penman–Monteith reference evapotranspiration (ET) , Priestley–Taylor reference ET and Hargreaves reference ET ). The method used to estimate net radiation may also play a large role. Although the results show that the ensemble mean provides best overall performance, the spread in ET is large (between 1.66 and 1.33 mm day-1).

All models provided SWE, while only six models provided SC. Total performance against the reference dataset was highest for ORCHIDEE, WaterGAP3 and LISFLOOD although the bias is fairly large for all models (see Fig. ). The phase difference seems to be influenced most by rather poor scores in the Himalaya. The total ensemble mean score for SWE is 0.67, which is lower than the highest model score of 0.74, suggesting that in this case the model mean should be used with care. However, GlobSnow has been shown to miss early season snow and SWE levels can drop too rapidly in spring . As most models show a later spring melt than GlobSnow it remains unclear if this is a model deficiency. A number of models show an unrealistic build-up of snow over time in Europe (HBV-SIMREG and PCR-GLOBWB), boreal North America (HTESSEL, HBV-SIMREG, PCR-GLOBWB), central Asia (HTESSEL, HBV-SIMREG) and Southeast Asia (HTESSEL, JULES, LISFLOOD). This may be caused by the fact that the models have been driven by a different dataset (with different temperature and radiation characteristics) than what they have been developed with. Snow cover fraction (SC) estimated by the models compares well to the IMS results. Here SURFEX-TRIP performs best but performance in general is much closer compared to the SWE results and the ensemble mean seems to provide a good estimate. Getting the phase correct in the Himalayan region seems to be the most challenging parameter for the models (see Fig. ).

Although current satellite-derived surface soil moisture products that cover a long period have a number of limitations , and are also not completely model-independent, they have been shown to capture intra- and inter-annual soil moisture variability, and this variability is not dependent on an external atmospheric forcing dataset. In addition, comparison with land surface models and in situ data showed good correlation . However, the long soil moisture record is not homogeneous because of sensor degradation as well as differences in sensor characteristics, algorithms and calibration . Therefore we used the period 2002–2012 only. Because only a limited number of models was able to provide surface soil moisture we have chosen to evaluate root zone soil moisture from the models with the remote-sensing product (see Table ). Although this may seem to be a large mismatch, the fact that we compare monthly average data should filter out the fast topsoil moisture fluctuation and make it more comparable. When comparing the results of the global models, the obtained differences are a result of the imperfections in the meteorological forcing, the model uncertainty – including parameterization and representative depth of the soil moisture – and the uncertainty in the soil moisture product. Global scores range from 0.45 for SWBM to 0.61 for HTESSEL and JULES with the ensemble mean score being very close to the best model at 0.60. Figure  shows the results for Australia and Southeast Asia. A recent study that used remotely sensed soil moisture to update the state of the PCR-GLOBWB hydrological model over a catchment in Australia showed significant improvement in simulated discharge after including the remotely sensed soil moisture, demonstrating that the hydrological models can benefit from the remotely sensed surface soil moisture. Conversely, showed that calibrating the SWBM against discharge only, yielded well represented soil moisture dynamics.

compared the set of model outputs described in this paper with discharge from 966 medium-sized catchments and demonstrated that the calibrated models showed best performance and that, on average, for the uncalibrated models the hydrological models performed better than the land surface models in snow-dominated regions. They also show that for example ORCHIDEE performs well in cold regions but tends to underestimate runoff in the other parts of the globe. This seems to be confirmed by the SC and SWE results for ORCHIDEE, and by its ET results that indicate that ORCHIDEE overestimates ET in high-ET regions. Combining the results of and the current results also shows that the models that have been calibrated on discharge do not necessarily give the best results for the variables we have used here. This is especially true for the HBV-SIMREG model which provides best results for discharge but has overall lowest scores when comparing it to the other datasets.

Table  summarizes the global water budget of all the models. The terrestrial runoff totals for all models apart from ORCHIDEE (where runoff also includes the excess water flowing off floodplains and irrigated areas: ) have been derived from the specific runoff per grid cell. As such the runoff into internal basins that do not drain into the ocean (endorheic basins) is included in the estimates and evaporation and abstractions from the routed water are not included (except for ORCHIDEE, where the evaporation in floodplains and abstraction for irrigation are included). The spread in runoff is fairly large (see Fig. ) and must originate from the difference in model concepts and parameterization (including the available energy partitioning) since the atmospherical forcing data used is identical for all models. Runoff is increasing after 1997 in all models. As can be seen from the top panel of Fig. , this is due to the elevated precipitation during the same period making more moisture available for both evaporation and runoff. As demonstrated by Fig.  the results plot closely to the precipitation minus runoff line with the LSMs generally showing more evapotranspiration and less runoff compared to the GHMs.

Table  presents the results of this study together with a selection of previous studies. Although results are not always directly comparable due to differences in land mask and techniques used, current results compare reasonably well with previous estimates. Yearly terrestrial runoff (excluding Antarctica and Greenland) from the 10 models ranges between 38 652 and 55 877 km3 yr-1 with an ensemble mean of 46 268 km3 yr-1. presented an optimized estimate of global terrestrial runoff of 45 900 km3 yr-1 ± 4400 km3 yr-1 for the period 2000–2010. Furthermore, the lower estimates compare well with findings from (44 200 ± 2660 km3 yr-1), while the ensemble mean compares well with the WATCH-based simulations of 49 680 km3 yr-1 and the results by (42 000 to 66 000 km3 yr-1), but are higher than estimates by (20 909 km3 yr-1, based on 430 basins estimated to cover 90 % of global runoff) and (37 288 km3 yr-1). The relatively high runoff in the estimates that rely on models, such as in this study, may in part be caused by the fact that they include small islands which are not represented in the gauge- and GRACE-based estimates.