COSMOS-Europe: a European network of cosmic-ray neutron soil moisture sensors

Climate change increases the occurrence and severity of droughts due to increasing temperatures, altered circulation patterns, and reduced snow occurrence. While Europe has suffered from drought events in the last decade unlike ever seen since the beginning of weather recordings, harmonized long-term datasets across the continent are needed to monitor change and support predictions. Here we present soil moisture data from 66 cosmic-ray neutron sensors (CRNSs) in Europe (COSMOS-Europe for short) covering recent drought events. The CRNS sites are distributed across Europe and cover all major land use types and climate zones in Europe. The raw neutron count data from the CRNS stations were provided by 24 research institutions and processed using state-of-the-art methods. The harmonized processing included correction of the raw neutron counts and a harmonized methodology for the conversion into soil moisture based on available in situ information. In addition, the uncertainty estimate is provided with the dataset, information that is particularly useful for remote sensing and modeling applications. This paper presents the current spatiotemporal coverage of CRNS stations in Europe and describes the protocols for data processing from raw measurements to consistent soil moisture products. The data of the presented COSMOS-Europe network open up a manifold of potential applications for environmental research, such as remote sensing data validation, trend analysis, or model assimilation. The dataset could be of particular importance for the analysis of extreme climatic events at the continental scale. Due its timely relevance in the scope of climate change in the recent years, we demonstrate this potential application with a brief analysis on the spatiotemporal soil moisture variability. The dataset, entitled “Dataset of COSMOS-Europe: A European network of Cosmic-Ray Neutron Soil Moisture Sensors”, is shared via Forschungszentrum Jülich: https://doi.org/10.34731/x9s3-kr48 (Bogena and Ney, 2021).

1 Introduction

The years 2003, 2010, 2015, and 2018 are considered as the most notable years of the 21st century in Europe in terms of summer drought and also witnessed numerous heat-related deaths (Stott et al., 2004; Ionita et al., 2017; Laaha et al., 2017; Schuldt et al., 2020; Sutanto et al., 2020) and extensive forest fires (Fink et al., 2004; Grumm, 2011; Turco et al., 2017). This has stimulated a debate on how changes in the occurrence and characteristics of drought are related to climatic variability (e.g., Hanel et al., 2018; Hisdal et al., 2001; Seneviratne et al., 2012; Sheffield et al., 2012). During the most recent heat wave in 2018, daily temperature anomalies reached up to 14 ^∘C in Scandinavia and central Europe and impacted the energy and carbon balance of European terrestrial ecosystems (Graf et al., 2020). This heat wave was exacerbated by a drought caused by a persistent circulation anomaly (Kornhuber et al., 2019), which additionally fostered unprecedented wildfires in Europe (e.g., Yiou et al., 2020). Recently, Humphrey et al. (2021) have shown that soil moisture variability explains 90 % of the interannual variability in global carbon uptake. The corresponding feedback between soil moisture and the atmosphere amplifies temperature and moisture anomalies and intensifies the direct effects of drought and soil water stress. In this respect, ground-based soil moisture measurements are indispensable to better understand the land-surface–atmosphere interactions leading to droughts and soil water stress.

Recent advances in measurement techniques, such as cosmic-ray neutron probes, allow continuous non-invasive soil moisture measurements that integrate over scales beyond the traditional point measurement (Zreda et al., 2012; Bogena et al., 2015; Andreasen et al., 2017b). In the 1950s it was discovered that neutron scattering could be used as a method of measuring soil moisture (e.g., Gardner and Kirkham, 1952), and this was to become the main means of quantifying water storage in soils for the next three decades. The neutron probe contains a radioactive source that generates fast neutrons that are decelerated by the hydrogen of the soil water to thermal neutrons, so that the detected thermal neutron count rate is closely related to the soil water content. Thanks to the pioneering work of Topp et al. (1980), from the 1980s the electromagnetic measurement technology became established for simple and continuous monitoring of soil moisture dynamics. As a result, neutron probes were hardly used anymore, and interest in neutron scattering in soils declined until the introduction of the cosmic-ray neutron measurement method (Zreda et al., 2008) generated renewed interest. Recently, neutron scattering is again considered one of the most promising soil moisture measurement techniques, as cosmic neutron sensors (CRNSs) provide non-invasive soil moisture at the field scale with an effective radius of 130 to 240 m and a penetration depth of 15 to 55 cm depending on soil wetness (Köhli et al., 2015; Schrön et al., 2017). In contrast to the classical active neutron probe, the CRNS is placed above ground and detects cosmogenic neutrons. The CRNS can be calibrated by comparing the neutron count rate with gravimetric soil moisture sampling data averaged over the CRNS footprint by a weighting function (Schrön et al., 2017). The CRNS shows excellent data acquisition reliability and can be applied also in vegetated areas with low to medium biomass such as cropped fields (Rivera Villarreyes et al., 2011; Franz et al., 2013) and forests (Bogena et al., 2013; Heidbüchel et al., 2016; Vather et al., 2020). During the last decade, several studies applied and progressed the CRNS technique both on stationary and mobile platforms up to the scale of square kilometers (Fersch et al., 2020; Schrön et al., 2018) and by monitoring stations installed in a broad variety of climate conditions, namely continental (e.g., Baatz et al., 2014), temperate (e.g., Evans et al., 2016), semi-arid (e.g., Zreda et al., 2012), and tropical (e.g., Hawdon et al., 2014). The advantages of the CRNS technique have promoted its application in various fields, such as hydrology (e.g., Dimitrova-Petrova et al., 2020a; Schattan et al., 2020), snow monitoring (e.g., Bogena et al., 2020; Schattan et al., 2017), precipitation monitoring (Franz et al., 2020), vegetation monitoring (e.g., Franz et al., 2013; Jakobi et al., 2018), validation of remote sensing products (e.g., Montzka et al., 2017; Duygu and Akyürek, 2019), land surface modeling (e.g., Shuttleworth et al., 2013; Baatz et al., 2017; Brunetti et al., 2019; Iwema et al., 2017, Patil et al., 2021), and agricultural management (e.g., Finkenbiner et al., 2018; Li et al., 2019).

According to Andreasen et al. (2017a), there are currently more than 200 stationary CRNSs operated worldwide, often as regional networks in hydrological observatories (e.g., Bogena et al., 2018; Kiese et al., 2018; Liu et al., 2018) or in entire countries (Zreda et al., 2012; Hawdon et al., 2014; Evans et al., 2016). This paper introduces the network of existing CRNS stations in Europe (COSMOS-Europe for short) and how the data are processed in a harmonized way. We present the current instrumentation and the protocols developed to process the raw measurements and how the CRNS stations have been recalibrated to derive soil moisture in a more consistent way. Based on the processed CRNS soil moisture time series, we then performed a brief analysis on the spatiotemporal occurrence of drought events in Europe.

2 Overview of the COSMOS-Europe sites

For the COSMOS-Europe dataset presented here, CRNS data from 66 sites in 12 European countries (in alphabetical order: Austria, Denmark, France, Germany, Greece, Italy, Norway, Poland, Spain, Switzerland, Turkey, United Kingdom) were collected. The geographical distribution and location of the COSMOS-Europe sites is shown in Fig. 1. The key environmental and soil-related physical properties at the sites are given in Table 1. The key physical and soil-related site properties relevant to CRNS processing are summarized in Table 2.

Figure 1 Locations of the COSMOS-Europe sites (the symbols show the climatic zone to which they belong) as well as sites, which are currently under construction or sites whose data we could not use.

Table 1 General information of the COSMOS-Europe sites (ordered based on latitude).

^a Derived from ERA5 data. ¹ Part of the groundwater and soil moisture monitoring network of the Norwegian water resources and energy directorate operated since 1998 and 1999, respectively. ² Part of the Danish Hydrological Observatory, HOBE (http://www.hobe.dk/, last access: 2 March 2022). Data are partly published by Andreasen et al. (2019; 2020) with more information on the field sites. ³ Part of the Moosbeere network operated by the University of Innsbruck. The Weisssee station is located on rocky terrain and not suited for soil moisture analysis. ⁴ Part of the Hydrological Open Air Laboratory (HOAL, https://hoal.hydrology.at/the-hoal, last access: 2 March 2022), which is a cooperation project between the Federal Agency for Water Management (BAW Petzenkirchen) and the Vienna University of Technology (TU Wien). More information about the site can be found in Blöschl et al. (2016). ⁵ Located at the Sheepdrove Organic Farm in the UK. More information about the site can be found in Iwema et al. (2021), Schrön et al. (2017), and Berthelin et al. (2020). ⁶ Part of the COSMOS-UK network operated by the UK Centre for Ecology & Hydrology. Further information about the sites and COSMOS-UK is presented in Cooper et al. (2021). ⁷ Part of the German Terrestrial Environmental Observatories (TERENO) network (https://www.tereno.net/, last access: 2 March 2022). ⁸ The CRNS is placed at the corner of three adjacent agricultural fields. Further information about the site can be found Dimitrova-Petrova et al. (2020a; 2020b; 2021). ⁹ Part of the ADAPTER (ADAPT TERrestrial systems) project (https://www.adapter-projekt.de/, last access: 2 March 2022; Ney et al., 2021). ¹⁰ Part of Alento Hydrological Observatory (AHO) in southern Italy established in 2016 by the University of Naples in cooperation with Forschungszentrum Jülich GmbH (Romano et al., 2018; Nasta et al., 2020). ¹¹ Due to a malfunction, the detector had to be replaced. To ensure a consistent time series, the neutron counts 1 year before and after the replacements were compared, and the time series after the replacement were adjusted accordingly. ¹² The Zugspitze sensor is located in the Schneefernerhaus (UFS) observatory and surrounded by rocky mountain terrain and not suited for soil moisture analysis. ¹³ Operated by the Water Resources Laboratory of Middle East Technical University. ¹⁴ Part of the Pinios Hydrologic Observatory (PHO) in central Greece established in 2017 by the Soil and Water Resources Institute, Hellenic Agricultural Organization “DEMETER”, in cooperation with Forschungszentrum Jülich GmbH (Pisinaras et al., 2018; Bogena et al., 2020). ¹⁵ Part of the MOSES (Modular Observation Solutions for Earth Systems) project (https://www.ufz.de/moses/, last access: 2 March 2022). ¹⁶ DWD – Deutscher Wetterdienst (2021).

Table 2 Physical quantities of the COSMOS-Europe sites.

The COSMOS-Europe sites cover eight climatic zones (following the Köppen–Geiger climate classification; Beck et al., 2018), with the vast majority of stations located in the humid continental climate zone (n=34) and in the temperate oceanic climate zone (n=21). The remaining sites are located in six further climate zones. According to site owner information, the majority of COSMOS-Europe sites are managed grassland (n=23) and cropland (n=23), while the remaining sites are covered by forest (n=7), forest clear-cut (n=1), shrubland (n=5), heathland (n=2), orchard/plantation (n=2), bare rock/glacier (n=1), moorland (n=1), and sparse vegetation (n=1).

The soils of the COSMOS-Europe sites range from organic soils with a high organic matter content (max: 0.173 g g⁻¹) to mineral soils with very low organic matter content (min: 0.004 g g⁻¹). This variability is also reflected in the wide range of soil porosities ranging from 0.365 to 0.841. Two of the sites, Weisssee and Zugspitze, are located in rocky, alpine terrain. The Weisssee data only show limited and short snow-free periods where soil moisture data are available and with high uncertainties due to the difficult soil sampling in that area. The data from Zugspitze are not used for soil moisture analysis due to the absence of soil but offer great potential for other hydrological studies, such as snow water equivalent monitoring.

The measurements of neutron count rates and corresponding correction data (i.e., atmospheric pressure and air humidity) at the COSMOS-Europe sites cover very different periods of time (cf. Fig. 2 and Table 1). The shortest time series comes from the Zerbst site, which was put into operation in late 2020. The longest time series from the Wuestebach1 site spans a period of approximately 10 years (mid-2011 to present). The average length of the observation periods of all sites is 5.7 years (±2.78).

Figure 2 Availability of atmospheric pressure, absolute humidity, and neutron count rates at the COSMOS-Europe sites (sorted by descending latitude). The dates of the local reference soil sampling for CRNS calibration are also shown. The timeline shown ends in June 2021, while data and calibration dates for some sensors extend to November 2021.

The geographic distribution of the COSMOS-Europe stations also reflects strong gradients of cutoff rigidities – a quantity describing the shielding of incoming cosmic-ray particles by Earth's geomagnetic field. Therefore, the dynamics and intensity of cosmic rays at stations in northern Europe are significantly higher than at stations further south. The cutoff rigidity ranges from 1.21 GeV for the Aas site in Norway to 8.37 for the Cakit Basin site in Turkey.

More than 50 additional sites are indicated in Fig. 1 (black cross) which are not specifically addressed in this paper. They either belong to other networks with dedicated data publications (e.g., COSMOS-UK or the intensive research experiment in Marquardt near Berlin), were installed just recently (e.g., Prague and northwest Germany), or refer to planned COSMOS locations in the near future (e.g., Finland and Ireland). There are even more stations across Europe that operate sub-snow cosmic-ray neutron detectors (Gugerli et al., 2019). Due to the slightly different measurement technique, the point-scale footprint, and the exclusive focus on snow monitoring, those sensors were not included in this paper and deserve dedicated articles.

3 Methods

Measured neutrons are a proxy for soil water content, but systematic factors and stochastic effects also influence the neutron signal. Research in the last decades has led to a profound understanding of these influencing factors and has facilitated a more accurate extraction of the soil moisture signal from the cosmic-ray neutron data. The processing framework is described below, while its technical implementation is supported by public tools and software libraries dedicated to CRNS research, e.g., Corny by Schrön (2021) and Crspy by Power et al. (2021).

3.1 Data pre-processing

In a first step, all datasets, i.e., raw neutron counts and supporting data, were aggregated to hourly time steps. Subsequently, following Zreda et al. (2008), a running 24 h average with a minimum of 12 measurements in the smoothing window was used to reduce the inherent noise of the raw neutron counts and to reduce the measurement uncertainty.

To ensure data consistency, the raw neutron counts were screened for data quality. Suspicious neutron count rates (N_raw) that fulfill one of the following conditions were flagged.

• Extreme single outliers: N_raw<50 or N_raw>10 000 counts per hour (cts h⁻¹).

• Positive suspicious peaks: N_raw>24 h moving average +2 times the standard deviation of the 24 h rolling sum.

• Negative suspicious peaks: N_raw<24 h moving average −2 times the standard deviation of the 24 h rolling sum.

Neutron count rates can be strongly affected by the presence of snow cover, resulting in inaccurate soil moisture measurements. Unfortunately, in most cases no additional snow measurements were available at the CRNS sites. Therefore, we used the ECMWF climate reanalysis data product ERA5-Land (Muñoz Sabater, 2019) to indicate snow cover events. For this, we flagged neutron count data when the 24 h moving average of the ERA5 SWE (snow water equivalent) product exceeded 1 mm.

To indicate unrealistically high values in the CRNS-derived soil moisture time series, we flagged values for soil moisture that were greater than local soil porosity. Because local measurements of soil porosity were not available, we estimated porosity using available information on bulk density and soil organic carbon content. We assumed that soil organic matter was 2 times the organic carbon content and assumed densities of 1.4 and 2.65 g cm⁻³ for the organic matter and the other soil minerals, respectively. Missing data on soil texture, porosity, and organic carbon were taken from the global raster-based soil dataset SoilGrids (Hengl et al., 2017).

It is important to note that the published dataset still includes the original and flagged data, while suspicious records were not included in the further data processing and analysis (see Fig. A5 for the used data flags). In this way, users can apply their own pre-processing techniques to the raw neutron count data. The final soil moisture product is cleaned from all negative influences to avoid inexperienced users using unrealistic soil moisture data.

The local air temperature, air humidity, and atmospheric pressure data needed for the correction of raw neutron counts often contained gaps due to measurement failure or due to removing suspicious data using max/min filters (see Fig. 2). These data gaps were filled with ERA5 data following the idea of Power et al. (2021). To ensure consistency of the data, linear regression models of the individual data time series were created to scale the ERA5 data to the local data prior to gap filling. Linear regression is necessary to compensate for differences in bias and slope, e.g., because due to the low spatial resolution of ERA5 (∼31 km), the average altitude, humidity, and atmospheric pressure for the ERA5 grid does not match those at the COSMOS-Europe site. These deviations occur especially in the high mountains due to strong elevation differences, e.g., for atmospheric pressure at the Leutasch site (see Fig. A2). The regression analysis showed that the ERA5 data mostly agreed well with the local measurements (Figs. A1 and A2), with mean correlations between ERA5 and local measurements of 0.95 for atmospheric pressure and 0.86 for absolute humidity. When the correlation coefficients for humidity and atmospheric pressure were less than 0.7 and 0.8, respectively, the local measurements were replaced entirely by ERA5 data to avoid inconsistencies in the gap-filled time series.

3.2 Correction of raw neutron counts

Variations of the incoming cosmic-ray intensity can have many causes, from galactic and solar disturbances to atmospheric and meteorological influences. Most of these anomalies are expected to change proportionally in every domain of the neutron energy spectrum and thus can be addressed by applying a set of correction factors,

$$\begin{array}{}\text{(1)}& N=N_{\mathrm{raw}}\cdot C_{\mathrm{p}}\cdot C_{\mathrm{h}}\cdot C_{\mathrm{inc}}\cdot C_{\mathrm{veg}}.\end{array}$$

The determination of the correction factors is explained in the following.

3.2.1 Atmospheric pressure correction

Since the cosmic-ray flux through the atmosphere is exponentially attenuated as a function of the traversed cumulative mass, measured neutron count rates can be normalized to standard atmospheric pressure by applying the standard pressure correction approach (Desilets and Zreda, 2003):

$$\begin{array}{}\text{(2)}& C_{\mathrm{p}}=e^{\mathit{\beta }\left(P-P_{\mathrm{0}}\right)},\end{array}$$

where C_p is atmospheric pressure correction factor, P₀ is the reference atmospheric pressure (1013.25 hPa), P is the actual atmospheric pressure, and β=0.0076 is the barometric coefficient that is related to the local mass attenuation length of neutrons in air. We also tested the application of regionally variable values for β according to Desilets and Zreda (2003) and Desilets et al. (2006), but found only negligible variations over Europe. However, future work should further investigate the influence of local β variability on the atmospheric pressure correction.

3.2.2 Air humidity correction

We accounted for the effect of atmospheric water vapor fluctuations on neutron count rate using the approach of Rosolem et al. (2013):

$$\begin{array}{}\text{(3)}& C_{\mathrm{h}}=\mathrm{1}+\mathit{\alpha }h,\end{array}$$

with α=0.0054 and h the absolute humidity (g m⁻³) measured at 2 m height.

3.2.3 Incoming neutron correction

The galactic cosmic radiation, or incoming radiation I(t), that penetrates the upper atmosphere varies in time mainly due to the well-known 11-year cycle of the solar activity. At high solar power (the solar maximum), the stronger solar magnetic field deflects a larger proportion of galactic particles away from Earth and reduces I(t). Conversely, during low solar activity (the solar minimum) the weaker solar magnetic field allows more galactic protons to enter the atmosphere increasing I(t). Shorter-term fluctuations have a similar effect on I(t) but with lower amplitude. Changes in the shape of the geomagnetic field, which occur on timescales from years to decades, are of secondary importance compared to temporal fluctuations of I(t). These temporal variations are measured locally with so-called neutron monitors (NM), which are sensitive to high-energy secondary neutrons (>20 MeV) but insensitive to local environmental factors (Simpson, 2000). The incoming radiation varies also spatially with strong gradients from the pole to the Equator, corresponding to the cutoff rigidity of the Earth's magnetic field. A worldwide network of NM stations provides near-real-time access to incoming cosmic-ray data (https://www.nmdb.eu/, last access: 2 March 2022). Assuming that the incoming radiation along the rigidity lines is similar, a nearby NM should be able to provide representative data for other places on Earth with similar cutoff rigidity R_cut. The local R_cut can be estimated for individual CRNS stations using approaches provided by Bütikofer et al. (2007). Since every detector comes with an individual efficiency, the value I(t) could be normalized with an arbitrary but constant reference I_ref, which we chose to be 150 cts h⁻¹. However, NM stations are rare, representing only a few latitudes and often not providing continuous signals over long periods of time. The NM at Jungfraujoch (Switzerland) is one of the few stations that provides reliable long-term data that can be used for COSMOS stations in Europe due its central location. Hence, scaling of the Jungfraujoch signal is needed to match the widespread distribution of COSMOS stations in Europe. According to Schrön et al. (2016), the intensity correction factor can be calculated as follows:

$$\begin{array}{}\text{(4)}& C_{\mathrm{inc}}=\left[\mathrm{1}+\mathit{\gamma }\left({\displaystyle \frac{I}{I_{\mathrm{ref}}}}-\mathrm{1}\right)\right]{}^{-\mathrm{1}},\end{array}$$

in which I is the count rate of incoming cosmic-ray neutrons of a neutron monitoring station, I_ref is the incoming count rate at an arbitrary time, and is an amplitude scaling factor to adjust for the mentioned geomagnetic effects. It depends on the cutoff rigidity of the local site and the neutron monitor used (see, e.g., Hawdon et al., 2014). For this paper, we use the approach from Hawdon et al. (2014) to bridge the regional difference of cutoff rigidities between the local site and the NM.

3.2.4 Biomass correction

Biomass can affect neutron count rates and should be considered when large temporal changes in biomass occur at a CRNS site. Therefore, we consider the biomass correction method proposed by Baatz et al. (2015) using the dry biomass B in kg m²:

$$\begin{array}{}\text{(5)}& C_{\mathrm{veg}}={\left[\mathrm{1}-\mathrm{0.009248}B\right]}^{-\mathrm{1}}.\end{array}$$

This correction was applied at the Wuestebach1 site, where a large change in biomass had occurred in the CRNS footprint area due to clear-cutting of a forest. For the other sites, there were no strong biomass changes or no detailed information on biomass changes was available. As soon as changes in the biomass occur or information for a site is available, these can be taken into account.

3.3 Sensor calibration

3.3.1 In situ reference soil data

For the calibration, we used in most cases available information on gravimetrically measured soil moisture from soil samples taken within the CRNS footprint. The soil samples were weighted vertically according to Schrön et al. (2017); i.e., for each sample at depth d and penetration depth D, we evaluate the weight in the representative sample volume (d₁ to d₂) to generate the profile average soil moisture:

$$\begin{array}{}\text{(6)}& {\mathit{\theta }}_{\mathrm{profile}}={\displaystyle \frac{\sum {\mathit{\theta }}_d{w}_d}{\sum w_d}},\mathrm{where}{w}_d=\underset{d_{\mathrm{1}}}{\overset{d_{\mathrm{2}}}{\int }}{w}_d\mathrm{d}d\propto W_{d_{\mathrm{1}}}-W_{d_{\mathrm{2}}}.\end{array}$$

In addition, we applied horizontal weighting of the vertically averaged in situ soil moisture according to Schrön et al. (2017). For this, regions of equal contribution (annuluses of 20 % quantiles) to the neutron signal were defined depending on the local conditions (i.e., atmospheric pressure, air humidity, average soil moisture) that influence the spatial sensitivity of the CRNS. All sampling points that fall within an annulus A are arithmetically averaged and thus receive the same weights, which are calculated according to the weighting scheme of Schrön et al. (2017). More specifically, θ_horiz is integrated over the entire domain to find the radii r₁ and r₂ that define the five annuluses A(r₁, r₂) within which all samples are equally averaged:

$$\begin{array}{}\text{(7)}& \begin{array}{rl}{\mathit{\theta }}_{\mathrm{horiz}}={\displaystyle \frac{\mathrm{1}}{\mathrm{5}}}\sum _{A=\mathrm{1}}^{\mathrm{5}}{\mathit{\theta }}_A,& \mathrm{where}{\mathit{\theta }}_A=〈{\mathit{\theta }}_r〉\forall r\in \left(r_{\mathrm{1}},r_{\mathrm{2}}\right)\\ & \mathrm{with}\underset{r_{\mathrm{1}}}{\overset{r_{\mathrm{2}}}{\int }}{W}_r\mathrm{d}r={\displaystyle \frac{A}{\mathrm{5}}}\underset{\mathrm{0}}{\overset{\mathrm{\infty }}{\int }}{W}_r\mathrm{d}r.\end{array}\end{array}$$

In particular, this method ensures that soil samples taken using the outdated COSMOS scheme (25, 75, 200 m), which assumed larger CRNS footprints, are not double weighted. Due to the long distances of the COSMOS sampling scheme, there may be no soil samples in one annulus. In this case, the samples in the next larger ring receive double the weight; i.e., the soil samples taken at 25 m distance are also representative for the soil moisture in the first annulus around the sensor. This problem does not arise for COSMOS-Europe sites sampled according to the revised weighting scheme of Schrön et al. (2017), as soil samples were also taken in the near field of the CRNS (i.e., 2–10 m distances). The in situ reference soil moisture of COSMOS-Europe sites and the weighted averages used for the CRNS soil moisture calibration are presented in Fig. A4.

3.3.2 Conversion of neutron count rate to soil moisture

To convert neutron count rates to soil moisture, we used the conventional relationship between neutrons and soil moisture initially introduced by Desilets et al. (2010). According to Köhli et al. (2021), it can be expressed in an equivalent but more unambiguous formulation with fewer parameters:

$$\begin{array}{}\text{(8)}& \mathit{\theta }\left(N\right)={\displaystyle \frac{\mathrm{0.0808}}{N/N_{\mathrm{0}}-\mathrm{0.372}}}-\mathrm{0.115}\equiv p_{\mathrm{0}}{\displaystyle \frac{\mathrm{1}-N/N_{\max }}{p_{\mathrm{1}}-N/N_{\max }}},\end{array}$$

where N_max is the maximum neutron flux under dry conditions which mainly depends on the individual detector sensitivity. Parameters $p_{\mathrm{0}}=-\mathrm{0.115}$, p₁=0.346, and $N_{\max }=\mathrm{1.075}\cdot N_{\mathrm{0}}$ can be derived from the parameters used so far in the Desilets equation. Calibrated values for N₀ of the COSMOS-Europe sites are given in Table 2.

Hydrogen in the organic matter as well as the lattice water content of soils affects how epithermal neutrons interact with the soil and thus affects the shape of the calibration function (Zreda et al., 2012). We accounted for this effect by fitting N_max of the calibration to the total soil water, which is the sum of the water equivalents of lattice water and organic matter and the gravimetrically measured reference soil moisture. The volumetric soil moisture is then obtained by subtracting the lattice and organic matter water from CRNS total soil moisture multiplied with the soil density. We averaged the soil property values, in case multiple calibration dates were available. In order to increase the signal-to-noise ratio of the neutron counts we applied a moving average with a window size of 24 h for the N_max calibration. It should be noted that a 24 h window is well suited for drought studies but may mask the signal from wetting events, which tend to occur much faster.

3.4 Soil moisture uncertainty

The statistical uncertainty of CRNS-derived soil moisture scales with the number of counts in a given period. However, this count rate is inversely related to soil moisture, so drier soils result in more accurate measurements (Desilets et al., 2010; Bogena et al., 2013). In addition, the size of the CRNS detector determines the count rate, i.e., a larger detector volume improves the count statistics and thus reduces the uncertainty of the soil moisture product (Weimar et al., 2020; Schrön et al., 2021). Different neutron detectors with different sizes and efficiencies are used in this study, so it is important to consider the CRNS-specific uncertainty (e.g., when using the data for validations). Due to the non-linearity of the neutron–soil-moisture relationship, the propagated uncertainty ±σ_θ is highly asymmetric (Iwema et al., 2021). For simplicity, it can be estimated by a symmetrical approximation approach suggested by Jakobi et al. (2020):

$$\begin{array}{}\text{(9)}& \begin{array}{rl}& \pm {\mathit{\sigma }}_{\mathit{\theta }}=\mathit{\theta }\left(N\right)-\mathit{\theta }\left(N\mp {\mathit{\sigma }}_N\right)\\ & {\mathit{\sigma }}_{\mathit{\theta }}\approx {\mathit{\sigma }}_N{\displaystyle \frac{p_{\mathrm{2}}{N}_{\max }}{{\left(N-p_{\mathrm{3}}{N}_{\max }\right)}^{\mathrm{4}}}}\\ & \sqrt{{\left(N-p_{\mathrm{3}}{N}_{\max }\right)}^{\mathrm{4}}+\mathrm{8}{\mathit{\sigma }}_N^{\mathrm{2}}{\left(N-p_{\mathrm{3}}{N}_{\max }\right)}^{\mathrm{2}}+\mathrm{15}{\mathit{\sigma }}_N^{\mathrm{4}}},\end{array}\end{array}$$

where the count rate N follows from the Desilets equation, ${\mathit{\sigma }}_N=C\sqrt{N_{\mathrm{raw}}}$ is its Gaussian uncertainty, and p₂=0.0752, p₃=0.346. We provide both the symmetric and asymmetric uncertainty of the CRNS-based soil moisture products in order to facilitate applications where only one of the two options can be used. It is important to note that these stochastic uncertainty estimates do not account for other (systematic) uncertainties, e.g., due to unconsidered biomass effects (Avery et al., 2016), N₀ calibration errors, and unconsidered variations in incoming neutron flux (Baroni et al., 2018), atmospheric pressure (Gugerli et al., 2019), and air humidity (Iwema et al., 2021). Please note that additional uncertainties may have occurred when filling gaps in the air pressure and humidity data with ERA5 data.

3.5 CRNS footprint radius and penetration depth

The footprint radius (i.e., R₈₆) was obtained as the 86 % cumulative contribution quantile of the weighting functions from Schrön et al. (2017). For this, we integrated the weights up to 600 m distance considering the influences of soil moisture (as the sum of the CRNS soil moisture, lattice water, and organic carbon), air humidity, and pressure. Subsequently, we obtained the average penetration depth (i.e., D₈₆) following Schrön et al. (2017), additionally considering the influence of soil bulk density.

3.6 Normalized quantiles of soil moisture

As suggested by Cooper et al. (2021), we use normalized quantiles to better indicate extreme soil moisture situations. First, the soil moisture values are normalized relative to the minimal and maximal observed soil moisture of the considered time series; i.e., the soil moisture values (θ) are scaled between 0 and 1 (p):

$$\begin{array}{}\text{(10)}& p=\left(\mathit{\theta }-{\mathit{\theta }}_{\min }\right){\mathit{\theta }}_{\max }/{\mathit{\theta }}_{\min },\end{array}$$

where θ_min is the minimum observed soil moisture and θ_max is the maximum observed soil moisture. Subsequently, p is used to obtain the quantile represented by each soil moisture value:

$$\begin{array}{}\text{(11)}& Q={\mathit{\theta }}_{\mathrm{sort}}\left(p\cdot n\right),\end{array}$$

where θ_sort represents the soil moisture values sorted in increasing order and n is the total number of soil moisture observations. From Q the median of all soil moisture values (θ_med) is subtracted, and the variance is scaled by dividing by the standard deviation (θ_SD) to obtain the normalized quantiles of soil moisture (Q_norm):

$$\begin{array}{}\text{(12)}& Q_{\mathrm{norm}}=\left(Q-{\mathit{\theta }}_{\mathrm{med}}\right)/{\mathit{\theta }}_{\mathrm{SD}}.\end{array}$$

Each Q_norm is then plotted against an observed value of the CRNS estimated soil moisture.

3.7 Implementation of the data processing

The raw neutron counts and meteorological data were converted into a uniform data structure (Fig. A5b) and stored in a database within the decentralized data infrastructure TEODOOR (TEreno Online Data repOsitORy, Kunkel et al., 2013). The data pre-processing, corrections, calibration, and uncertainty estimation were implemented in the programming language Python. These scripts were applied to the raw data stored in the database using Node-RED, a graphical tool for deploying workflows (Node-RED, 2021). Node-RED offers the possibility to connect different data flows in a simple way, using so-called nodes. Each node has a defined and unique task. When data are transmitted to a node, the node can process these data and then transmit them to the next node. In this way, different corrections or other implementations in the data post-processing can be added or removed individually. As an interface for accessing the data in the TERENO database, the Sensor Observation Service (SOS) of Open Geospatial Consortium was used. Here, the data are processed by a separate proxy that forwards the requests to a virtual Python environment. In the last step, the processed data were written back directly to the database via email or SOS.

The raw data as well as the processed data are accessible via the TERENO Data Discovery Portal (DDP) at http://www.tereno.net (last access: 2 March 2022). The data portal enables the query, visualization, and access to data and metadata of the stations presented in this paper. Additionally, detailed information on each CRNS station is provided in the metadata (Fig. A5a) and can be retrieved from the data portal.

4 Results and discussion

4.1 Spatiotemporal occurrence of drought events in Europe

The provision of a continental-scale dataset on soil moisture dynamics opens up numerous possibilities for analysis, especially with respect to large-scale climatic and hydrological applications. In the following, we present first analyses on the spatiotemporal occurrence of drought events in Europe based on the processed time series of CRNS soil moisture.

Figure 3 visualizes the results of the CRNS soil moisture processing for the COSMOS-Europe sites. The CRNS soil moisture (left subplot) shows strong temporal variations as well as large differences between the COSMOS-Europe sites. Due to these strong variations in CRNS soil moisture, similarities in the absolute values are difficult to discern, e.g., the impact of large-scale drought events on CRNS soil moisture. Therefore, following the approach of Cooper et al. (2021), Fig. 3 also presents the normalized quantiles of CRNS soil moisture (right subplot) to better indicate extreme soil moisture situations, i.e., to better distinguish between “normal low” soil moisture and “extremely low” soil moisture. In this way, the widespread impacts on the recent drought events of 2018, 2019, and 2020 on CRNS soil moisture in Europe become more apparent. The 2018 drought, in particular, is clearly visible with pronounced negative values in the normalized soil moisture quantiles across all latitudes, indicating that the whole of Europe was affected by the drought.

Figure 3 Time series of CRNS soil moisture (a) and normalized quantiles of CRNS soil moisture (b) of the COSMOS-Europe sites ordered from north to south according to latitude (unrealistic soil moisture values are excluded, i.e., larger than porosity). The timeline shown ends in June 2021, while data and calibration dates for some sensors extend to November 2021.

In the following, we explore if CRNS soil moisture information can be a valuable basis for more accurate assessment of the uniqueness and potential impacts of drought events at regional to continental scales. In Fig. 4 the monthly mean CRNS soil moisture of all COSMOS-Europe sites since 2011 are presented, along with the spatial mean and SD (upper subplot).

Figure 4 Time series of monthly mean CRNS soil moisture (gray lines in panel a) and normalized quantiles of CRNS soil moisture of the COSMOS-Europe sites (black dots in panels b–d). For three exemplary days during recent drought periods in Europe (8 August 2018, 7 July 2019, and 2 June 2020), the normalized quantiles are highlighted and differentiated by climate zone. These days were selected as they exhibited the lowest hourly soil moisture during the drought events. The mean of normalized quantiles of CRNS soil moisture for these days is also shown.

Despite the different time series lengths, the seasonal variations in soil moisture can be clearly seen. We selected three drought events to examine differences in soil moisture between sites, with all data for the period of record presented as normalized quantiles of soil moisture for each site (Fig. 4, lower subplot). It is evident that sites even within the same climate zone with broadly similar weather patterns can have very different ranges and extremes of soil moisture.

This finding confirms results by Cooper et al. (2021) for the United Kingdom, who in particular suggested heterogeneity of soil properties as an explanation for the variabilities, and results of Dong and Ochsner (2018), who found that soil moisture at the regional scale is more controlled by soil texture than precipitation. However, when comparing the three events, it becomes evident that 2018 had more locations with pronounced extremes and that these occurred predominantly in the climate zone Cfb, while 2018 was not notably different from other drought years in the monthly soil moisture averages shown in Fig. 4 (upper subplot). This demonstrates again the advantage of normalized soil moisture quantiles for a more in-depth analysis of extreme events.

Finally, we investigated whether the CRNS data allow us to draw conclusions about longer-term trends in soil moisture in Europe. For this, we contrasted monthly mean soil moisture from 2014–2017 with monthly mean soil moisture from 2018 to 2021 in Fig. 5, using the 26 sites fully covering this period. From Fig. 5, it is evident that as of 2018, soil moisture was lower not only in the summer months but throughout the year. Although the considered soil moisture data cover only 7 years, they can be considered as an indicator of the magnitude and direction of the trend in soil water supply that Europe can expect as climate change progresses.

Figure 5 Comparison of monthly mean soil moisture from 2014 to 2017 and monthly mean soil moisture from 2018 to 2021 using 26 COSMOS-Europe sites that cover these periods.

4.2 How representative and accessible is the soil moisture data?

The representativeness of the individual stations for the depicted land use type and geographical location is relevant, especially with regard to the validation of large-scale model applications of remote sensing products, which usually show coarser spatial resolutions and correspondingly “averaged” representation of site properties (e.g., Colliander et al., 2017; Montzka et al., 2020). In a few cases, the COSMOS-Europe sites represent larger site heterogeneity. This is reflected in particular in the resulting larger variability of soil moisture in the in situ calibration data measured for the individual sites (see Fig. A4). An example is the CRNS station at the Fuerstensee site in Germany (Rasche et al., 2021). The footprint represented by the CRNS measurement at this site includes a sand lens in the center of the footprint, which is surrounded by peat soils. The resulting heterogeneity, particularly of soil properties such as bulk density, soil organic matter content, and lattice water, challenges the harmonized data processing applied for COSMOS-Europe and leads to greater uncertainty in the derived soil moisture product, which is not easy to quantify.

While aspects such as soil heterogeneities or varying land use can be of great importance for local or CRNS methodological questions, this is rather an obstacle for large-scale questions. Especially with regard to the future development of the European network of CRNS stations, attention should be paid to the selection of sites that guarantee a high representativeness and homogeneity. With respect to the use of COSMOS-Europe data to derive conclusions about continental-scale trends in soil moisture, it is decisive that the network ensures the most representative coverage of key environmental and geographic gradients throughout Europe (e.g., altitude, climate, landforms, geology). This is currently the case only to a limited extent (see also Fig. 1). The clear majority of stations are concentrated in central Europe, while Scandinavia, eastern Europe, or the Mediterranean region in particular are covered by only very few stations. This limits the interpretability of the data, especially with regard to comparisons between different climate zones.

Another important question in this context is whether observations at a limited number of points can provide regional improvements in the prediction of hydrologic states and fluxes. For example, Baatz et al. (2017) assimilated measured soil moisture data from a CRNS network into the area-differentiated land surface model CLM 4.5 (Oleson et al., 2013) and showed that updating states and hydraulic parameters leads to better regional hydrologic predictions. This indicates that the COSMOS-Europe data could be beneficial for model applications at the continental scale despite the limited coverage in some areas of Europe, i.e., even though the measurements are not area-wide such as remote sensing data.

Furthermore, at present only a low number of CRNS stations are automatically transferring neutron count data to the TERENO database that hosts the COSMOS-Europe data. However, a near-real-time availability of the data would be necessary in particular for the use of the data for the improvement of flood models, e.g., in context of the European Flood Awareness System (EFAS, Smith et al., 2016). Efforts should be made in the future to equip more stations with automatic data transfer capabilities to enable rapid transfer of neutron counts to the TERENO database. Here, the implemented automated routines for data pre-processing, correction, and neutron counts to soil moisture allow for immediate provision of COSMOS-Europe data products.

Finally, we would like to point out that there are still alternative data products to some of the CRNS stations used here, which were processed using less comprehensive methods. Although our processing methods closely follow the commonly used method presented by Zreda et al. (2012), there are some important differences as we make use of the latest CRNS research findings from the last 10 years. For example, the US COSMOS database (http://cosmos.hwr.arizona.edu/, last access: 2 March 2022) does not contain corrections for air humidity, which were suggested by Rosolem et al. (2013) and Köhli et al. (2021). Another main difference of our processing scheme is the weighting of the in situ calibration data according to neutron transport theory (Schrön et al., 2017). We therefore expect soil moisture deviations that can be substantial depending on the date of previous processing schemes.

5 Data availability

The dataset, entitled “Dataset of COSMOS-Europe: A European network of Cosmic-Ray Neutron Soil Moisture Sensors”, is stored in a common data format and shared via Forschungszentrum Jülich (https://teodoor.icg.kfa-juelich.de/ibg3butt/ibg.butt.download?FileIdentifier=519e0691-7eb3-4351-aff5-c0a0335933ab, last access: 24 September 2021); https://doi.org/10.34731/x9s3-kr48 (Bogena and Ney, 2021).

Potential users can also access the data of the individual CRNS stations in a dedicated section for COSMOS-Europe in the TERENO data portal TEODOOR at https://ddp.tereno.net/ddp/dispatch?searchparams=keywords-Cosmic%20Ray (last access: 2 March 2022). Here, metadata information about the stations (e.g., site owner) as well as the raw data and the processed data products can be accessed. Please note that downloads will only be made possible via a token and are provided with a disclaimer with the terms of use.

6 Conclusions and outlook

In this data paper, we present soil moisture data from 65 CRNS stations that are distributed across Europe and cover all major land use types and climate zones. The raw neutron count data from the CRNS stations were processed using state-of-the-art methods in a harmonized way including correction of the raw neutron counts and conversion into soil moisture based on available in situ information. In addition, information on the data uncertainty is added to the dataset – information that is particularly useful for remote sensing and modeling applications. It should be noted that the sites have individual heterogeneous conditions, which cannot always be adequately reflected by a standard processing scheme. In addition, the data processing used in this work represents the state of the art, but this may change as a result of future research. We therefore provide raw data and will update the published dataset with an incremental version number if new processing procedures become accepted in the future.

We show that the COSMOS-Europe dataset enables a good representation of the magnitude and distribution of drought events. However, so far, only the central part of Europe is particularly well covered by COSMOS-Europe, while there are still large gaps in the peripheral areas of Europe. The density of COSMOS stations in Europe is still not sufficient to completely represent soil moisture patterns across all parts of the continent. Thus, future efforts should invest in higher observational coverage. One emphasis in the further development of COSMOS-Europe must be to convince countries to put CRNS stations into operation that do not yet operate CRNS stations or hardly any. In addition, efforts should be made in the future to equip more stations with automatic data transfer capabilities to enable near-real-time accessibility of soil moisture information, e.g., to support flood forecasting. The data presented here can be used for a manifold of hydrological applications, such as drought assessment, flood risk assessment, and snow water estimation.

Similar to COSMOS-Europe, several other large-scale COSMOS networks already exist in the USA, Australia, and India. The obvious next step is to build on the methods developed in this study to create a global network of continental COSMOS networks, similar to the FLUXNET initiative for eddy covariance measurements of land–atmosphere exchange fluxes (FLUXNET, 2021) or the International Soil Moisture Network (ISMN) that provides in situ soil moisture data from 2842 stations worldwide (Dorigo et al., 2021). Initial networking efforts in this direction have already been undertaken.

Appendix A

Figure A1 The correlations between air humidity from local measurements and ERA5 for the COSMOS-Europe sites.

Figure A2 The correlations between atmospheric pressure from local measurements and ERA5 for the COSMOS-Europe sites.

Figure A3 Detected unrealistic CRNS soil moisture estimates due the presence of snow at the site (i.e., times of snow water equivalent from ERA5 larger than 1 mm) and soil moisture values exceeding the local soil porosity.

Figure A4 Soil profiles of the in situ calibration data for the COSMOS-Europe sites. The weighted average soil moisture values are also shown. The varying colors indicate the different sampling dates.

Figure A5 Data structure in the TERENO Data Discovery Portal (DDP). Each station comprises metadata (a) with detailed site information and two time series. One time series contains the raw CRNS data, the meteorological data, and the processed data with the associated diagnostics (b). The second time series provides the raw calibration data (c).