Two sets of monthly precipitation normals (long-term averages) were used to create the CHPclim. The first set was a collection of 27 453 monthly station averages obtained from the Agromet Group of the Food and Agriculture Organization of the United Nations (FAO). This extensive collection has a fairly detailed level of representation in many typically data-sparse regions, but suffers from a limitation. The FAO database does not provide the period of record used to calculate the long-term averages, although most observations roughly correspond to averages over the 1950s through the 1980s. This data set, therefore, was augmented with 20 591 station climate normals taken from version two of the Global Historical Climate Network (GHCN) (Peterson and Vose, 1997). We compensated for the FAO database's varied coverage in time by supplementing it with averages from a less dense but more temporally consistent information source – the GHCN. The more extensive FAO normals were used to build the preliminary climate surfaces (as described below in Sect. 3). The differences between this surface and GHCN 1980–2009 averages were then estimated and interpolated, and then used to adjust the final monthly surfaces to a 1980–2009 time period.

Monthly means of four satellite products were evaluated as potential background climate surfaces: Tropical Rainfall Measuring Mission (TRMM) 2B31 microwave precipitation estimates (Huffman et al., 2007), the Climate Prediction Center morphing method (CMORPH) microwave-plus-infrared based precipitation estimates (Joyce et al., 2004), monthly mean geostationary infrared (IR) brightness temperatures (Janowiak et al., 2001), and Land Surface Temperature (LST) estimates (Wan, 2008). The TRMM and CMORPH precipitation estimates are based primarily on passive microwave observations from meteorological satellites in asynchronous orbits. The monthly mean infrared brightness temperatures, on the other hand, are derived from a combination of multiple geostationary weather satellites. The LST estimates are derived from multispectral observations from Moderate Resolution Imaging Spectrometers (MODIS) aboard the Terra and Aqua satellites. The LST fields are global, while the CMORPH, TRMM, and IR brightness temperatures span 60∘ N/S. For each month, for all available years (typically ∼ 2001–2010), the satellite data were averaged. All four products were convolved to a common 0.05∘ grid. A fifth predictor was created based on the average of the CMORPH and TRMM precipitation fields.

Mean 0.05∘ elevation, compound topographic index, flow accumulation, aspect, and slope were calculated from global 30 arcseconds GTOPO30 elevation grids following the methodology developed for the HYDRO1K (Verdin and Greenlee, 1996). While the utility of all the topographic fields was explored, only elevation and slope were used in the final analysis because they proved to be the most robust predictors. Latitude and longitude were also included as potential predictor variables.

Our process relies heavily on local regressions between our target variable and background field. We begin by explaining the bivariate standardized case of this process, which corresponds to a localized correlation. At a certain location we can sample a number of points and background variables that fall within a certain distance (dmax) and calculate their distance weighted (localized) correlation. The localized correlation process finds a set of n neighboring points (within dmax), and estimates their weighted correlation. This study uses a cubic function of the distance (d) and a user-defined, regionally variable, maximum distance (dmax). wd= 0,d>dmaxwd=1-d/dmax33d≤dmax.

These weights are then used to estimate a localized correlation. rx,y=n-1-1(∑i=1nw(di))-1∑i=1nw(di)(xi-x¯)σx-1(yi-y¯)σy-1.

The localized correlation (rx, y) at some location (x, y) corresponds with the standardized cross-product of the neighboring points, weighted by their distance. This process can be used to generate correlation maps (Fig. 1). Typically, the direct physical relationship between the station normals and a satellite field, such TRMM/CMORPH precipitation, results in a stronger correlation pattern than that which is produced by an indirect physiographic indicator such as elevation. Figure 1 provides an example of this by contrasting the local correlations between station precipitation, elevation and TRMM/CMORPH precipitation.

The core of the CHG climatology modeling process is based on a series of local regressions between in situ observations and spatially continuous predictor fields. For each location, a set of neighboring observations is obtained, and a regression model constructed using weighted least squares, with the weight of each observation determined by its distance from the regression centroid (Eq. 1). For each region and month, a grid of center points is defined on a regular 1∘ grid over land-only locations. Figure 2 shows the modeling regions. At each center-point, station values within the radius (dmax) are collected, and a regression model is fit based on weights determined by Eq. (1). The dmax values are defined individually for each model region, varying from 650 km for the larger or data-sparse regions (e.g. Australia, northwest Asia) to 300 km for Central America and the Galapagos.

For each modeling region and month, regression models were determined through a combination of automated regression subset selection and visual inspection of the output. In some cases, visual inspection indicated that a combination of statistically powerful predictors produced obvious artifacts. In these cases, the selection pool was reduced by hand. Based on the boundaries of the interpolation window, certain predictors were omitted (TRMM, CMORPH, IR) because the satellite range did not extend northward or southward enough for these areas.

Following the MWR modeling procedure, at-station anomalies (the arithmetic difference between the FAO station normals and the nearest 0.05∘ regression estimate) are calculated and interpolated using a modified IDW interpolation procedure. For each 0.05∘ grid cell, the cube of inverse distances is used to produce a weighted average of the surrounding station residuals, r. This value is then modified based on a local interpolation radius, dIDW and the distance to the closest neighboring station (dmin). r∗=1-dmindmindtextIDWdIDWr.

This simple thresholding procedure forces the interpolated residual field to relax towards zero, based on the distance to the closest station. The dmin values were defined by modeling region, and ranged from 350 to 100 km, based on station density. All tiles were allowed to overlap with their neighbors, and locations within these areas of overlap were blended based on weights that were linear functions of the distances from tile edges. This helped to produce smooth transitions from tile to tile.

In the final stage, for each month, the regional tiles are composited on a global 0.05∘ grid and compared with 1980–2009 GHCN climate normals. The ratio of the GHCN and gridded climatology is calculated at each station location. These ratios are capped between 0.3 and 3.0, and interpolated to a 0.05∘ grid for each month. The values were capped to limit the potential influence of poor station data. A modified IDW procedure, similar to Eq. (3), is used, but instead of relaxing to zero, the interpolation is forced to a ratio of 1 (no change) as the distance to the minimum neighbor reaches dIDW. This ratio grid is multiplied against the sum of the MWR and interpolated residuals, producing the final CHG Climatology field.

Selection bias can inflate the estimated accuracy of statistical estimation procedures, producing artificial skill (Michaelsen, 1987). To limit such inflation, this study uses cross-validation. This technique removes 10 % of the station data, fits the model using the remaining 90 % of the values, and evaluates the accuracy for the withheld locations. This process is repeated ten times, eventually withholding all of the data, to produce a robust estimate of the model accuracy.

As additional validation, high quality climatology data sets were obtained for five focus regions: Afghanistan, Colombia, Ethiopia, Mexico, and the Sahel region of western Africa (Senegal, Burkina Faso, Mali, Niger and Chad). Means, spatial R2 values, mean bias errors (MBE [mm]), mean absolute errors (MAE [mm]), percent MBE, and percent MAE statistics were evaluated. These regions (as opposed to the continental United States or Europe) were chosen to represent challenging estimation domains.

Figure 2 shows the best predictor for each individual modeling region and the FAO station locations. For regions between 60∘ N and 60∘ S, the combined CMORPH and TRMM field tended to be the most useful predictor. The TRMM-only precipitation was selected, however, for southern Africa. Regions beyond 60∘ N and 60∘ S could not be modeled with the TRMM or CMORPH means. These regions were generally best fit with LST, slope, or elevations from a digital elevation model (DEM). Figures 3 and 4 show the proportion of modeled cross-validated variance for the MWR and interpolated residuals components for each of the modeling regions. These results are averaged across the 12 months. For most regions, the MWR accounted for over 80 % of the total variance. The interpolated residuals typically accounted for another 10–25 %. Most regions of the globe had average monthly percent errors of between 15 and 25 % (Fig. 5). Figure 6 shows monthly mean CHPclim precipitation fields. As discussed later, these seem generally quite similar, in most places, to the GPCC M V2015, the CRU CL v2.0, and the Worldclim version 1.4 release 3 products. The blending of the overlapping tiles creates generally smooth transitions from tile to tile. These products will be compared more closely later in this paper.

We next present results from our validation studies for Afghanistan, Colombia, Ethiopia, Mexico, and the Sahel (Senegal, Burkina Faso, Mali, Niger, and Chad). In each case, additional high-quality gauge data were obtained from national meteorological agencies (Table 1). These data were screened, and only values not in the FAO or GHCN archive were retained. Table 1 summarizes the number of independent stations and presents the monthly validation statistics, averaged across all 12 months. For each validation station, the closest CHPclim, CRU, or Worldclim grid cell was extracted. The CHPclim percent biases were substantially smaller in magnitude than the CRU or Worldclim biases, ranging between -2 to +5 %, as compared to -28 to +16 % (CRU) or -16 to 0 % (Worldclim) or -1 to -17 % (GPCC). Note that the GPCC gauge observations were corrected for systematic under-catch errors (Becker et al., 2013; Schneider et al., 2014). While all the climatologies did well in regions with a large number of stations (e.g. Mexico and Colombia), CHPclim's performance was substantially better in data-sparse areas like the Sahel, Ethiopia, and Afghanistan. Averaged across these study regions, the CHPclim/CRU/Worldclim/GPCC data sets had overall mean absolute error (MAE) values of 16, 26, 20 and 20 mm month-1, respectively. The average spatial R2 values for the four climatologies were 0.77 (CHPclim), 0.58 (CRU), 0.67 (Worldclim), and 0.51 (GPCC). Overall, the CHPclim compared favorably to the CRU, Worldclim and GPCC data sets.

Plotting the monthly validation statistics provides more temporal information. Figure 7 shows monthly time series of the MAE values for each region and for each set of climatological estimates. In Afghanistan, data were only obtained for the rainy season. The low spatial correlations with the CRU and Worldclim estimates (Table 1) translate into high MAE scores (Fig. 7). In Colombia, the spatial R2 (Table 1) and MAE time series of the CHPclim and Worldclim are similar – both perform well. In Ethiopia, the Worldclim and CRU MAE peak in concert with the seasonal rainfall maxima, while the CHPclim values remain substantially lower. This pattern is recreated for the Sahel and, to a lesser extent, for Mexico. We postulate that the CHPclim performance benefits from the fact that satellite precipitation estimates do a good job of representing heavy convection in these countries during the heart of the precipitation season. Conversely, the thin plate spline fitting procedure, combined with low gauge density in Ethiopia and the Sahel, may make it difficult to statistically represent precipitation gradients in these countries, degrading the performance of the CRU and Worldclim climatologies. Thin plate splines fit polynomial surfaces through point data, creating a generalized surface fit to latitude, longitude, and elevation. The suitability of this fitting process may be problematic when the density of the gauge data is very low. Later in our paper we compare different climate products over Ethiopia.

Figure 8 shows similar time series for the spatial R2 statistics. In Afghanistan, Ethiopia, and the Sahel, the CHPclim appears substantially better at representing spatial gradient information. In Colombia and Mexico, CHPclim and Worldclim performance is similar. This may relate to the number of climate normals available in each region (cf. Fig. 2). In Colombia and Mexico, relatively dense gauge networks result in similar Worldclim and CHPclim performance. In regions with fewer stations, the correlation structure of the satellite precipitation data (Fig. 1) probably helps boost the relative performance of CHPclim.

Here we briefly examine differences between quasi-global total annual precipitation from the CHPclim, GPCC M V2015, CRU CL 2.0 and Worldclim version 1.4 release 3 (Fig. 9) and their global and continental averages (Table 2). The left hand panels show differences between the CHPclim and the three other products. The largest differences appear over the north half of South America, where annual precipitation is very high (Fig. 6). These differences may arise from the local influence of the satellite rainfall fields, which are well correlated with station observations in this region (Fig. 1). Note that the GPCC, CRU, and Worldclim also vary substantially amongst themselves in this area. In Europe, northern Asia, North America, and Australia, the differences are fairly limited, most likely due to the high station density in these regions. There are large differences near the Himalayas. The CHPclim appears to be producing more precipitation across the Himalayan plateau and less precipitation on the south-facing mountain slopes. More research will be required to evaluate if this is appropriate or not. CHPclim also appears to be substantially drier over some parts of Africa. A recent study in Mozambique (Toté et al., 2015) of the Climate Hazards group Infrared Precipitation with Stations (CHIRPS, Funk et al., 2014b), which is based on the CHPclim, found low bias over that country. Stations in Africa tend to be biased towards wet locations, and the use of satellite fields as guides to interpolation may help limit this bias. We explore this idea in more detail in the next section, which focuses on an Ethiopia test case.

Before proceeding to that analysis, we note that the global (excluding Antarctica) and continental averages from our four products are in quite close agreement (Table 2), even in Africa. The two outlier's appear to be the GPCC M V2015 averages for Asia (688 mm) and for the globe (880 mm). The global GPCC M V2015 value of 880 mm is close to the 850 mm figure reported in Schneider et al. (2014). The discrepancy between the GPCC results and the other products is likely due to the way they corrected for systematic under-catch by the rainfall gauges. The CHPclim does not incorporate this correction which increases precipitation observations based on estimated under catch values. The global (excluding Antarctica) total annual rainfall values, expressed in “units” of global precipitation of 103 km3 yr-1 as in Trenberth et al. (2007) agree quite well with that study's reported values (110 units CRU; 112 units GPCP). We found the CHPclim precipitation resulted 120 units. This difference may relate to CHPclim's interpolation procedure in northern South America and the Maritime Continent where the CHPclim is wetter (Fig. 9, Table 2), perhaps because of guidance provided by satellite observations (Fig. 1).

In February of 2015 one of the co-authors led a rainfall gridding workshop in Addis Ababa, in collaboration with lead scientists from the Ethiopian National Meteorological Agency (NMA). This workshop used the GeoCLIM tool to blend CHIRPS satellite rainfall estimates with 208 quality-controlled gauge observations (Figs. 10 and 11, top left) to generate monthly 1981–2014 grids of precipitation. In this section we compare the 1981–2014 average of these blended CHIRPS/NMA station data to the CHPclim, GPCC, CRU and Worldclim data sets. We acknowledge that since the CHPclim is used in the CHIRPS as a background climatology the NMA and CHPclim data sets are not completely independent. Nonetheless, the 35 years of 208 NMA rain gauge observations have not been included in the CHPclim, and hence provide a valuable validation data set, especially within the areas with good gauge density.

Figure 10 shows the mean 1981–2014 annual rainfall totals based on the gridded NMA data, and similar maps from the CHPclim, GPCC M V2015, CRU CL 2.0, and Worldclim version 1.4 release 3. Also shown are elevation, annual totals of CMORPH/TRMM precipitation and annual average MODIS LST. These fields were used in the CHPclim modeling process. Annual mean MODIS Normalized Difference Vegetation Index (NDVI) values are also shown as an independent proxy for moisture availability. All the precipitation products and the NDVI agree on the broad patterns of spatial rainfall variability, which are extreme. The wettest regions receive more than 2 m of rainfall each year while the driest receive less than 200 mm. The CMORPH/TRMM satellite observations seem to capture these dry areas well – with no ground data at all, i.e. the brown areas in the CMORPH/TRMM agree quite closely with the NMA validation data. The CMORPH/TRMM fields delineate dry area effectively. Within wet areas, the discriminatory power of the satellite observations seems to diminish, indicating (incorrectly) that northwest Ethiopia is as wet as southwest Ethiopia. The similarity between the completely independent NDVI and NMA/CHPclim fields is quite compelling. Many subtle features, such as the humid highlands in north-central, east-central, and southeastern Ethiopia appear well demarcated by these precipitation fields. These seem fairly well captured by the Worldclim and CRU as well.

Note that there are important differences between, on one hand, the elevation and similar LST field and, on the other, the NMA/CHPclim precipitation and NDVI mean fields. While there are certainly some important correspondences, there are also critical differences, such as in north-central Ethiopia which is high and cool, but dry. Conversely, northwest Ethiopia is relatively wet, but relatively low. There are times and locations when elevation is a poor indicator of mean precipitation.

Figure 11 shows the differences from NMA validation data. Also shown, to support analysis, are the NMA mean precipitation and elevation data. Purple lines have been drawn showing transects plotted in Fig. 12. The CHPclim follows the NMA climatology closely. The GPCC, CRU, and Worldclim all exhibit substantial (>| 300 mm |) deviations, with the Worldclim performing substantially better than the GPCC and CRU. This helps to confirm the visual impression from Fig. 10 that the Worldclim data follows the NMA data quite closely. The GPCC, CRU and Worldclim all underestimate precipitation in the blue regions in the northwest and southwest of these maps, which are relatively low areas. The CMORPH/TRMM finds rainfall in these areas (Fig. 10), and the CHPclim MBE in these areas is quite modest (Fig. 11). Conversely, dark brown areas in the bottom panels of Fig. 11 denote areas where rainfall is substantially overestimated in the GPCC, CRU, and Worldclim. This appears to be of gravest concern in the center and center-east of the country, which has high elevations and extremely steep rainfall gradients. While not perfect, the CMORPH/TRMM (Fig. 10) seems to capture these gradients with reasonable fidelity, and building on these gradients produces a CHPclim with low bias in these areas.

We explore this topic more fully in Fig. 12, which shows transects of our data sets at 10 and 7∘ N. We have multiplied the NDVI data by 1500 and divided the elevation data by 5 to facilitate visualization. Begin by noting in the top two panels the similarities between the mean NMA data, the CMORPH/TRMM, and the NDVI. This reinforces the utility of the TRMM/CMORPH, and that the NMA fields are an effective representation of the “true” climatology. The CRU and Worldclim seem to follow the NMA transect quite well, with some substantial deviations shown in the bottom panels. Some of these errors appear to coincide with areas having extreme elevation changes, such as 36.5∘ E, 37.5∘ E and 40∘ E at 10∘ N. At 37∘ E, 7∘ N, the CRU, GPCC and Worldclim substantially underestimate rainfall. The CHPclim, assisted by the CMORPH/TRMM, which is quite wet in this region, captures the rainfall well. In the eastern part of the country, where we find the largest percent discrepancies, we find overestimates at 41∘ E, 10∘ N and 41.5∘ E, 7∘ N. Estimates of rainfall gradients in these poorly instrumented regions are very difficult based on just station data. The CMORPH/TRMM, however, seems to capture these gradients well, and the CHPclim builds on this local gradient information.