SGD-SM 2.0: an improved seamless global daily soil moisture long-term dataset from 2002 to 2022

The drawbacks of low-coverage rate in global land inevitably exist in satellite-based daily soil moisture products because of the satellite orbit covering scopes and the limitations of soil moisture retrieving models. To solve this issue, Zhang et al. (2021a) generated seamless global daily soil moisture (SGD-SM 1.0) products for the years 2013–2019. Nevertheless, there are still several shortages in SGD-SM 1.0 products, especially in temporal range, sudden extreme weather conditions and sequential time-series information. In this work, we develop an improved seamless global daily soil moisture (SGD-SM 2.0) dataset for the years 2002–2022, to overcome the above-mentioned shortages. The SGD-SM 2.0 dataset uses three sensors, i.e. AMSR-E, AMSR2 and WindSat. Global daily precipitation products are fused into the proposed reconstructing model. We propose an integrated long short-term memory convolutional neural network (LSTM-CNN) to fill the gaps and missing regions in daily soil moisture products. In situ validation and time-series validation testify to the reconstructing accuracy and availability of SGD-SM 2.0 (R: 0.672, RMSE: 0.096, MAE: 0.078). The time-series curves of the improved SGD-SM 2.0 are consistent with the original daily time-series soil moisture and precipitation distribution. Compared with SGD-SM 1.0, the improved SGD-SM 2.0 outperforms on reconstructing accuracy and time-series consistency. The SGD-SM 2.0 products are recorded in https://doi.org/10.5281/zenodo.6041561 (Zhang et al., 2022).

1 Introduction

Surface soil moisture acts as a significant part of global hydrology and meteorology, especially for forecasting drought and flood disasters (Wigneron et al., 1999; Long et al., 2014; Brocca et al., 2018). In recent years, satellite-based soil moisture retrieving data have rapidly progressed in both global and daily monitoring (Shi et al., 2006; Dorigo et al., 2012, 2021; Al Bitar et al., 2017), e.g. AMSR-E, AMSR2 and WindSat global daily soil moisture products and so on (Fan et al., 2004). These quantitative products have been widely utilised for global and long-term hydrological analysis and forecasting (Chen et al., 2021; Todd-Brown et al., 2022).

However, because of the limitations of soil moisture retrieving models and satellite orbital covering scopes, the obtained daily soil moisture products are fragmentary and incomplete (Shi et al., 2002; Enenkel et al., 2016; Meng et al., 2021). As shown in Fig. 1a and b, plenty of gap regions exist in these soil moisture products. Actually, the land coverage rate is only approximately 20 % to 80 % in daily AMSR-E, AMSR2 and WindSat quantitative products (Long et al., 2019; Bogena et al., 2022).

Figure 1 Daily soil moisture products of AMSR-E and WindSat.

To settle this adverse effect for global soil moisture applications, most works adopted the temporal averaging operation such as monthly, quarterly or yearly averaging (Schaffitel et al., 2020; Guevara et al., 2021; Wang et al., 2021). This strategy could usually acquire full-coverage soil moisture products via averaging abundant daily products. Nevertheless, the temporal averaging operation is also a two-edged sword. Firstly, it directly replaces daily temporal resolution with low-frequency temporal resolution (Rebel et al., 2012; Long et al., 2020; Zhang et al., 2021), which greatly lowers the utilisation of daily soil moisture products. Secondly, a temporal averaging operation disregards the specific spatial distribution of daily products and neglects the sequential time-series changing characteristic (Zeng et al., 2015a; Wang et al., 2021; Zhan et al., 2015). In other words, a monthly, quarterly or yearly averaging strategy degrades the original characteristics for daily soil moisture products.

To address this issue, Zhang et al. (2021a) generated a seamless, global, daily soil moisture (named SGD-SM 1.0) dataset from 2013 to 2019. The spatial resolution is denoted as a 0.25^∘ grid (about 25 km). The SGD-SM 1.0 dataset relies on the deep spatio-temporal partial convolutional model to fill the gaps or missing regions in daily soil moisture products. Thereafter, three validations are performed to verify the reliability of SGD-SM 1.0 products. Relevant quantitative indexes (R, RMSE and MAE) and results demonstrate that SGD-SM 1.0 products can be extended for global, daily and full-coverage soil moisture measurements (Zhang et al., 2021a).

The SGD-SM 1.0 dataset maintains the original high-frequency daily temporal resolution and effectively enhances the utilisation of global daily soil moisture products. However, several weaknesses and limitations also exist in SGD-SM 1.0. Based on SGD-SM 1.0 and the above-mentioned considerations, we develop an improved seamless global daily soil moisture (SGD-SM 2.0) dataset for the years 2002–2022 in this work. Compared with SGD-SM 1.0, the main improvements and contributions of SGD-SM 2.0 are listed below:

• The SGD-SM 1.0 dataset only uses a single sensor (AMSR2), and the temporal range is insufficient with only 7 years. While global soil moisture analysis and applications generally need longer-term and more multi-sensors products, the application range of SGD-SM 1.0 is still limited. Compared with SGD-SM 1.0, SGD-SM 2.0 uses three passive microwave sensors (AMSR-E, WindSat and AMSR2). The temporal range of SGD-SM 2.0 is extended to 20 years from 2002 to 2022. The application scope of SGD-SM 2.0 could be enlarged through these long-term soil moisture products.

• The SGD-SM 1.0 dataset ignores the daily extreme weather conditions. If sudden precipitation occurs in 1 d, SGD-SM 1.0 usually performs poor under this scenario. The main reason is that SGD-SM 1.0 relies on the internal spatio-temporal correlation, which does not consider the external environmental factors. Compared with SGD-SM 1.0, SGD-SM 2.0 introduces the global daily precipitation products into the reconstructing framework. Through fusing auxiliary precipitation data, SGD-SM 2.0 could lead in the daily extreme weather information for gap-filling.

• Although SGD-SM 1.0 employs 3-D partial convolutional neural network (CNN) to exploit both spatial and temporal features, it is still insufficient for utilising sequential time-series information. For daily soil moisture products, how to effectively reconstruct gaps and missing regions through interrelated temporal information is significant. Compared with SGD-SM 1.0, SGD-SM 2.0 develops an integrated long short-term memory convolutional neural network (LSTM-CNN) to fill the gaps and missing regions in these daily products. The proposed LSTM-CNN model could simultaneously utilise recurrent time-series information and spatial information.

• Compared with SGD-SM 1.0 products, SGD-SM 2.0 products outperform on R (0.688), RMSE (0.094) and MAE (0.077). In addition, the time-series curves of the improved SGD-SM 2.0 products are more consistent with the original daily time-series soil moisture values. Benefiting from the data fusion of daily precipitation information, the proposed LSTM model can extract time-series features for filling the gaps and missing regions in daily soil moisture products. Therefore, SGD-SM 2.0 can be utilised effectively for global hydrology monitoring and analysis at fine (daily) temporal resolution.

The outline of this work is arranged as follows. Section 2 provides a description of products and data used in this work. Section 3 gives the methodology of the proposed reconstructing framework for SGD-SM 2.0. Section 4 lists the experimental results of SGD-SM 2.0 products. Section 5 discusses the comparisons between SGD-SM 1.0 and SGD-SM 2.0, especially regarding reconstructing accuracy and time-series consistency. Finally, the conclusion and outlook are summarised in Sect. 6.

2 Products and data description

In this work, we simultaneously fuse satellite-based soil moisture products and precipitation products to generate the SGD-SM 2.0 dataset. The in situ soil moisture sites are employed to validate the reconstructing precision of SGD-SM 2.0. These in situ data are downloaded from the International Soil Moisture Network (ISMN). Detailed descriptions are listed below.

2.1 Satellite-based soil moisture products

The AMSR-E, AMSR2 and WindSat global daily soil moisture products are utilised from 2002 to 2022. These three sensors are onboarded at the Aqua satellite, GCOM-W1 and Coriolis satellite, respectively (Nepal et al., 2021). The AMSR-E, AMSR2 and WindSat products are all passive sensors for soil moisture retrieving. The spatial resolution is a 0.25^∘ grid (about 25 km) in all these products, as depicted in Fig. 1a–c. The retrieving model adopts the land parameter retrieval model (LPRM) for AMSR-E, WindSat and AMSR2 products (McColl et al., 2017). We select the descending orbit (nighttime) and 6.9 GHz band for all these soil moisture products. These datasets are all recorded in the GES DISC website. These three products provide the original information for using SGD-SM 2.0. The proposed reconstructing model acquires the gap masks and relies on the valid spatio-temporal soil moisture information from these three products to fill the gaps and missing regions.

The time-series range of the AMSR-E sensor starts from 19 June 2002 and ends on 4 October 2011 (Njoku et al., 2003; Shi et al., 2008). The time-series range of the WindSat sensor starts from 1 February 2003 and ends on 2 August 2012. The time-series range of the AMSR2 sensor starts from 3 July 2012 and continues to the current date (Zeng et al., 2020). In consideration of the low-coverage rate in the WindSat dataset, we only use WindSat global daily products from 5 October 2011 to 2 July 2012 for acquiring sequential daily products. These recorded AMSR-E, WindSat and AMSR2 global daily products are all employed as the initial input of the proposed LSTM-CNN model for generating SGD-SM 2.0 products. The daily coverage rate curves of these three global quantitative products are depicted in Fig. 2a–c, respectively.

Figure 2 Daily coverage rate curves of AMSR-E, WindSat and AMSR2 soil moisture products in 2002, 2012, and 2021, respectively.

2.2 Precipitation products

Precipitation usually has a high correlation with soil moisture in the corresponding regions (Pellarin et al., 2009; Brocca et al., 2014a; Sun and Fu, 2021). Therefore, we fuse the precipitation products into the proposed SGD-SM 2.0 dataset to improve the reconstructing accuracy. The Integrated Multi-satellitE Retrievals for GPM (IMERG) global daily precipitation V6 products are employed for the years 2002–2022 (Massari et al., 2020). These precipitation products are derived from multiple precipitation-relevant satellite passive microwave sensors, as portrayed in Fig. 3a. The spatial resolution is denoted as a 0.1^∘ grid (about 10 km) in IMERG level 3 global daily final precipitation products. To keep the uniformity with soil moisture products, the spatial downsampling operation is carried out for the original IMERG precipitation products from 0.1 to 0.25^∘. We then normalise these precipitation values via linear transformation for use in the reconstructing model. These precipitation products were all downloaded from GES DISC (Brocca et al., 2019; Berg et al., 2021; Škrk et al., 2021).

2.3 In situ soil moisture data

In situ soil moisture sites are significant for testifying to the satellite-based products (Brocca et al., 2014b; Gruber et al., 2020). These sites provide high-precision surface soil moisture values. Relying on in situ data, the quantitative indexes could be derived for the proposed SGD-SM 2.0 dataset. The ISMN unites global in situ surface data, which have been widely applied for hydrology and soil moisture validation (Dorigo et al., 2011, 2013, 2021; Wigneron et al., 2013). We select 124 stations from the ISMN from 2002 to 2022 and match them with corresponding soil moisture products in SGD-SM 2.0 (Zhang et al., 2020). The selected criteria include three points:

1. The in situ soil moisture sites are downloadable through the given website.

2. The in situ soil moisture sites are continuous for the long-term observation, at least 1 year.

3. The spatial distribution of these in situ sites covers various continents, land use and soil types.

The spatial distribution of these selected in situ data is displayed in Fig. 3b. These in situ soil moisture data are public and can be downloaded from https://ismn.geo.tuwien.ac.at/en/ (last access: 13 January 2022). The in situ validation results of SGD-SM 2.0 and the reconstructing accuracy comparisons with SGD-SM 1.0 are provided in Sects. 4.2 and 5.1, respectively.

Figure 3 IMERG global daily precipitation and selected in situ data.

3 Methodology

The schematic representation of the proposed work is depicted in Fig. 4. Different from SGD-SM 1.0, we simultaneously fuse global daily precipitation products with global daily soil moisture products into SGD-SM 2.0. An integrated long short-term memory convolutional neural network (LSTM-CNN) reconstructing model is developed to fill the gaps and missing regions in global daily soil moisture products. Finally, we recursively generate the seamless daily soil moisture products in the SGD-SM 2.0 dataset. Detailed descriptions of the proposed LSTM-CNN reconstructing model, training and optimisation are stated below.

Figure 4 Schematic representation of the proposed framework to generate SGD-SM 2.0 products.

3.1 LSTM-CNN reconstructing model

As shown in Fig. 4, the original global daily soil moisture product on date T (AMSR-E, WindSat or AMSR2) and its corresponding global daily precipitation product are utilised as the input data of the proposed framework. Firstly, the precipitation data on date T are transformed as the vector value P_T through a full-connected (FC in Fig. 4) layer. We employ the partial convolutional neural network (Partial CNN in Fig. 4) to extract the spatial feature of soil moisture product on date T. Different from the common CNN (Yuan et al., 2019), partial CNN can effectively acquire the spatial information within valid regions and eliminate the invalid information within gaps or soil moisture missing regions (Zhang et al., 2018a). We applied a partial CNN for generating the SGD-SM 1.0 dataset. Due to its effectiveness on incomplete soil moisture products, the partial CNN is also used in this work for generating the SGD-SM 2.0 dataset. The formula of a partial CNN in this work is determined as follows:

$$\begin{array}{}\text{(1)}& {{\mathbf{S}}^{\prime }}_{(m,n)}=\left\{\begin{array}{cc}{\mathbf{W}}^{\top }({\mathbf{S}}_{(m,n)}\otimes {\mathbf{M}}_{(m,n)})\frac{{∥{\mathbf{1}}_{(m,n)}∥}_{\mathrm{1}}}{{∥{\mathbf{M}}_{(m,n)}∥}_{\mathrm{1}}}+b,& {∥{\mathbf{M}}_{(m,n)}∥}_{\mathrm{1}}\ne \mathrm{0}\\ \mathrm{0},& \mathrm{otherwise}\end{array}\right.,\end{array}$$

where M denotes the mask of its corresponding soil moisture product S; 0 and 1 refer to the invalid and valid points in the mask M. The trainable weighted and offset arguments in the partial CNN are denoted by W and b, respectively; and ⊗ represents the dot product operation to exclude the invalid information in gaps or missing regions through mask data. Subsequently, the current mask M needs to be regenerated under the below paradigm: if the partial convolution can generate at least one valid value of the output result, we need to mark this location as a valid value in the new masks. The regenerated mask formula is defined as follows (Zhang et al., 2020):

$$\begin{array}{}\text{(2)}& {{\mathbf{M}}^{\prime }}_{(m,n)}=\left\{\begin{array}{cc}{\mathbf{L}}_{(m,n)},& {∥{\mathbf{M}}_{(m,n)}∥}_{\mathrm{1}}\ne \mathrm{0}\\ \mathrm{0},& \mathrm{other}\end{array}\right.,\end{array}$$

where L_(m,n) stands for Earth land in position (m, n). It should be noted that the Earth land mask includes six continents and neglects all regions of Antarctica and most regions of Greenland. The main reason is that these omitted regions are perennially covered with snowy or frozen land (Zhao et al., 2021).

After four partial CNN layers in Fig. 4, an FC layer is also acted on in the feature maps of the soil moisture product, with the result of vector value S_T:

$$\begin{array}{}\text{(3)}& S_T=\mathrm{FC}\left({{\mathbf{S}}^{\prime }}_{\mathrm{4}}\right)s.t.P_T=\mathrm{FC}\left({\mathbf{P}}_T\right).\end{array}$$

Then, the two vectors, S_T and P_T, of soil moisture and precipitation products are simultaneously imported into the LSTM model in Fig. 4. The architecture of the LSTM model within the proposed framework is displayed in Fig. 5.

Figure 5 Structure of the LSTM model in the proposed framework.

As depicted in Fig. 5, soil moisture information S_T, corresponding precipitation information P_T, previous long-term memory information C_T−1, and previous short-term memory information h_T−1 are simultaneously imported into the LSTM model. The output values in the LSTM are the regenerated soil moisture information ${S^{\prime }}_T$, current long-term memory information C_T, and current short-term memory information h_T. It should be noted that current LSTM information on date T is the previous LSTM information on next date T+1, respectively. For memory information h₀ and C₀, these vectors are initialised with zero elements. The LSTM is composed of three gates, i.e. oblivious gate, input gate and output gate to control the memory information state (Zhang et al., 2021b).

• Oblivious gate. This gate determines which information needs to be discarded in the short-term memory state. It is carried out by the sigmoid unit σ between the soil moisture information S_T and previous long-term memory information C_T−1:

  $$\begin{array}{}\text{(4)}& f_T=\mathit{\sigma }(W_f\cdot [h_{T-\mathrm{1}},S_T]+b_f),\end{array}$$

  where the sigmoid unit σ is defined below:

  $$\begin{array}{}\text{(5)}& \mathit{\sigma }\left(a\right)={\displaystyle \frac{\mathrm{1}}{\mathrm{1}+e^{-a}}}.\end{array}$$

  In the sigmoid unit σ, 0 means fail and 1 means pass for current information. Through checking the soil moisture information S_T and previous long-term memory information C_T−1, it generates a vector between 0 and 1. This variable determines which information is retained or discarded in the short-term memory state.

• Input gate. This gate determines which new information is added to the long-term memory state. Firstly, we use h_T−1 and S_T to determine which information needs to be updated through the sigmoid operation in Eq. (6):

  $$\begin{array}{}\text{(6)}& i_T=\mathit{\sigma }(W_i\cdot [h_{T-\mathrm{1}},S_T]+b_i).\end{array}$$

  Then the new candidate long-term memory information, ${\stackrel{\mathrm{̃}}{C}}_T$ is generated through the tanh unit in Eq. (7):

  $$\begin{array}{}\text{(7)}& {\stackrel{\mathrm{̃}}{C}}_T=\tanh (W_C\cdot [h_{T-\mathrm{1}},S_T]+b_C),\end{array}$$

  where the tanh unit is defined as:

  $$\begin{array}{}\text{(8)}& \tanh \left(a\right)={\displaystyle \frac{e^a-e^{-a}}{e^a+e^{-a}}}.\end{array}$$

• Output gate. In this gate, we need to output current long-term memory information C_T from previous C_T−1 and candidate long-term memory information ${\stackrel{\mathrm{̃}}{C}}_T$:

  $$\begin{array}{}\text{(9)}& C_T=f_T\otimes C_{T-\mathrm{1}}+i_T\otimes {\stackrel{\mathrm{̃}}{C}}_T.\end{array}$$

After updating current long-term memory information C_T, the regenerated soil moisture information ${S^{\prime }}_T$ is output through previous short-term memory h_T−1, soil moisture information S_T, and corresponding precipitation information P_T:

$$\begin{array}{}\text{(10)}& {S^{\prime }}_T=\mathit{\sigma }\left(W_{S^{\prime }}\right[h_{T-\mathrm{1}},S_T]+b_{S^{\prime }})\otimes \tanh \left(P_T\right).\end{array}$$

Besides, we need to output current short-term memory information h_T for the next date T+1 as follows:

$$\begin{array}{}\text{(11)}& h_T=\tanh (o_T\otimes C_T).\end{array}$$

Later, the regenerated soil moisture information ${S^{\prime }}_T$ is transformed by an FC layer in the right of Fig. 4. Thereafter, four partial CNN layers are performed to generate the final SGD-SM 2.0 product on date T. Through the consecutive time-series strategy, we recursively reconstruct the daily soil moisture products in SGD-SM 2.0.

3.2 Training and optimisation

To generate the reliable and high-precision SGD-SM 2.0 dataset, the way in which to train and optimise the proposed LSTM-CNN model is extremely crucial in this work. The training stage needs huge numbers of sample labels to optimise the trainable parameters in the proposed partial CNN and LSTM in Figs. 4 and 5, respectively. The sample labels adopt a patch-selecting strategy. We select sequential time-series daily soil moisture patches with k=7 in the reconstructing framework. The spatial sizes of these 7 d soil moisture patches are all set as 40×40. These time-series 7 d soil moisture patches are all complete, without gaps or data missing regions from the original soil moisture products. Then, we randomly select 30 000 mask patches with the spatial size of 40×40. Each soil moisture patch is simulated with missing regions via these mask patches. In this way, we acquire 30 000 training samples from the original 2002–2022 soil moisture products. Each training sample includes four variables: the simulated 7 d soil moisture patches, the complete 7 d soil moisture patches, the corresponding mask patches, and the corresponding precipitation patches. These variables are simultaneously imported into the LSTM-CNN reconstructing model, as shown in Fig. 4.

For the partial CNN in the proposed framework, we set the convolutional filter size as 3×3 in all the partial CNN layers (Xiao et al., 2022a). The last partial CNN layer outputs just one feature map and the other partial CNN layers output 64 feature maps (Xiao et al., 2022b). The rectified linear unit (ReLU) is utilised after each partial CNN layer. For the LSTM model in the proposed framework, we set the dimension of long and short-term memory vectors C_T and h_T as 2048.

For the network optimisation, we adopt the same strategy with the global–local function (Zhang et al., 2021a) in SGD-SM 1.0. The global soil moisture uniformity and local soil moisture heterogeneity are both taken into consideration in the proposed LSTM-CNN reconstructing model. Different from SGD-SM 1.0, we simultaneously fill the gaps and missing regions in the time-series 7 d soil moisture patches. Detailed definitions of the global–local function are determined as follows:

$$\begin{array}{}\text{(12)}& \begin{array}{rl}\mathit{\xi }& (\mathbf{W},b,W_{f,i,C,S^{\prime }},b_{f,i,C,S^{\prime }})\\ & =\sum _{T=\mathrm{1}}^k\left({∥(\mathrm{1}-{\mathbf{M}}_T)\otimes ({\mathbf{S}}_T^{\mathrm{rec}}-{\mathbf{S}}_T^{\mathrm{ori}})∥}_{\mathrm{2}}^{\mathrm{2}}\right.\\ & \left.+{∥{\mathbf{M}}_{\mathrm{L}}\otimes ({\mathbf{S}}_T^{\mathrm{rec}}-{\mathbf{S}}_T^{\mathrm{ori}})∥}_{\mathrm{2}}^{\mathrm{2}}\right),\end{array}\end{array}$$

where M_L represents the global land mask (including six continents and neglecting all regions of Antarctica and most regions of Greenland); α stands for the balancing parameter to equilibrate the local loss and global loss (Zhang et al., 2020). Empirically, this ratio is fixed as 0.1 in the training and optimisation stage.

In terms of the hyperparameters and operations of the proposed framework, related explanations are listed below. The batch size of the LSTM-CNN reconstructing model is set as 128 (Zhang et al., 2018a). The whole epoch number is confirmed, determined as 500 (1 epoch denotes that all the samples in the training set have been utilised for the neural network optimisation at one time). The inceptive learning rate is started at 0.005 (Zhang et al., 2018b). It gradually decreases through multiplying a damping factor (equal to 0.5) every 100 epochs (Zhang et al., 2019). On software configuration, the LSTM-CNN model is carried out on the PyTorch 1.8.1 framework. We use Python 3.7 language, PyCharm platform and the Windows 10 environment to generate seamless global daily soil moisture products. On hardware configuration, we employ a NVIDIA Titan X (Pascal) GPU, Inter E5-2609v3 CPU, and 16 GB DDR4 RAM to execute the proposed LSTM-CNN model.

4 Experiments and validations

The released SGD-SM 2.0 products are recorded in https://doi.org/10.5281/zenodo.6041561 (Zhang et al., 2022). The SGD-SM 2.0 dataset starts from 23 June 2002 and ends on 5 February 2022. The initial and reconstructing global daily soil moisture products have been stored with individual NetCDF4 (*.nc) documents. Because part of the daily soil moisture products is missing at GES DISC, these products are also neglected in the proposed SGD-SM 2.0 dataset (7115 files). In this section, the experimental results of the SGD-SM 2.0 dataset are given in Sect. 4.1. Later, we carry out the in situ validation and time-series validation of SGD-SM 2.0 in Sect. 4.2 and 4.3, respectively.

4.1 Experimental results

As shown in Figs. 6 and 7, the original SM and SGD-SM 2.0 results are given on 10, 20 and 30 September 2002 and on 10, 20 and 30 June 2020, respectively.

Figure 6 Original SM and proposed SGD-SM 2.0 results on 10, 20 and 30 September 2002.

For comparison purposes, the left lines are the original global daily products and the right lines are the reconstructed SGD-SM 2.0 products in Figs. 6 and 7. It should be noted that we neglect all regions in Antarctica and most regions in Greenland because of the perpetual frozen soil. Clearly, gaps and missing regions are filled through the proposed framework in Sect. 3.

Figure 7 Original SM and proposed SGD-SM 2.0 results on 10, 20 and 30 June 2020.

From the spatial perspective, the proposed SGD-SM 2.0 dataset performs both global soil moisture uniformity and local soil moisture heterogeneity in Figs. 6 and 7. It ensures the spatial consistency, especially for the gap regions with the adjoining soil moisture regions. Beyond that, the reconstructed regions in SGD-SM 2.0 do not reflect distinct patch or border effects. This also testifies to the powerful ability of the partial CNN in the proposed framework, which can effectively exclude the invalid information in gaps or missing soil moisture regions.

From the temporal perspective, the proposed SGD-SM 2.0 dataset utilises the complementary and sequential time-series soil moisture information. By fusing global daily precipitation products, SGD-SM 2.0 can consider the sporadic extreme weather conditions for a single day. In addition, by means of the LSTM model, the consistent temporal information can be recovered and preserved in Figs. 6 and 7.

Figure 8 Scatters of six in situ sites (horizontal coordinate refers to the in situ data; vertical coordinate denotes the reconstructing data).

4.2 In situ validation

In situ validation is the most reliable method to measure the accuracy and availability of the proposed SGD-SM 2.0 dataset (Walker et al., 2004; Draper et al., 2009; Zeng et al., 2015b). In this work, we choose 124 in situ surface (0–5 cm depth) soil moisture sites from the ISMN, as shown in Fig. 3b. The selected in situ values are limited from 2002 to 2022. We match the hourly in situ values with the descending products. In consideration of validation reliability, we choose the two neighbouring in situ values that correspond with the observation time of soil moisture products. We then average them as the ground-truth data.

As portrayed in Fig. 8, the scatters of six in situ soil moisture sites (marked as blue circles in Fig. 3b: 42.537^∘ N, 72.171^∘ W; 0.282^∘ N, 36.866^∘ E; 48.141^∘ N, 15.171^∘ E; 14.159^∘ S, 131.388^∘ E; 21.617^∘ S, 47.632^∘ W; 31.369^∘ N, 91.899^∘ E) are displayed to demonstrate the reconstructing accuracy of SGD-SM 2.0. The horizontal coordinate refers to in situ data. Accordingly, the vertical coordinate denotes reconstructing data in gaps or missing soil moisture regions. The time range is limited from 2002 to 2022. The R indicators of these sites are varied from 0.658 to 0.769 in Fig. 8a–f. The RMSE and MAE indicators of these sites are varied from 0.023 to 0.144 and from 0.021 to 0.128, respectively in Fig. 8a–f.

Table 1 Comparisons between the original SM and SGD-SM 2.0 products (from 2002 to 2022) in the 124 selected in situ sites.

In all 124 selected in situ sites, Table 1 compares the original products with the SGD-SM 2.0 products. The average evaluation indicators (R, RMSE, and MAE) of the original SM (SGD-SM 2.0) products are 0.679 (0.672), 0.094 (0.096) and 0.075 (0.078), respectively. Generally, the precision of SGD-SM 2.0 products performs similar with incipient products. The diversities of those indicators are few between the original and reconstructed SGD-SM 2.0 products in Table 1. To a certain extent, in situ validation testifies to the reconstructed accuracy and validity of the SGD-SM 2.0 products. These 124 selected soil moisture stations from the ISMN (2002–2022) are shown in Table 2 for validating SGD-SM 2.0. Besides, basic information about the representative in situ soil moisture sites (taking partial sites as examples, including COSMOS, SD-DEM, SMOS-CBR, SCAN, PBO, USCRN and OZNET networks) is listed in Table 3. As the example, it includes the name of the station, country, longitude/latitude, main land use, lattice water, and soil organic carbon.

Table 2 124 selected soil moisture stations from the ISMN (2002–2022) for validating SGD-SM 2.0.

Table 3 Basic information about the in situ soil moisture sites (taking partial sites as examples).

4.3 Time-series validation

Long-term daily soil moisture products usually reflect typical time-series continuity (Wang et al., 2022; Seneviratne et al., 2010). Therefore, we can utilise this characteristic to validate the reliability of SGD-SM 2.0 products. As listed in Figs. 9 and 10, 2 time-series daily original/SGD-SM 2.0 results of 2003–2018 and 2005–2020, are given in the location (10.125^∘ S, 42.625^∘ W) and the location (38.375^∘ N, 117.125^∘ E), respectively. The blue dots refer to the existing valid values in Figs. 9 and 10. The red dots stand for the SGD-SM 2.0 values in Figs. 9 and 10, which also represents the invalid gaps or missing soil moisture regions. The vertical coordinate denotes the percent of soil moisture product in original and SGD-SM 2.0 products. The horizontal coordinate denotes the annual date number between 2003 and 2020.

As depicted in Figs. 9a–d and 10a–d, a majority of the reconstructed SGD-SM 2.0 values (in invalid gaps or missing soil moisture regions) can distinctly embody the time-series continuity. In the two locations of different years, original soil moisture values and corresponding adjacent SGD-SM 2.0 values perform fore-and-aft consistency. If current valid soil moisture values behave high or low, their neighbouring SGD-SM 2.0 values also accord with them in Figs. 9 and 10. This time-series validation manifests the reliability of the proposed framework and validity of our improved SGD-SM 2.0 products. Generally, the proposed SGD-SM 2.0 products are able to ensure the time-series continuity in daily temporal resolution. This point is greatly important for reconstructing long-term products. Benefiting from the utilisation of temporal information, the proposed LSTM model can extract and transmit time-series features for filling the gaps and missing data regions in daily soil moisture products. Therefore, SGD-SM 2.0 can be effectively applied for global hydrology monitoring analysis at fine-temporal scale, rather than the traditional monthly or yearly averaging operation. The former preserves the original daily temporal resolution, while the latter sacrifices this daily temporal resolution. This validation exactly demonstrates the above significance of the proposed SGD-SM 2.0 dataset.

Figure 9 Time-series daily original/SGD-SM 2.0 results of the location (10.125^∘ S, 42.625^∘ W) in 2003, 2008, 2013 and 2018.

Figure 10 Time-series daily original/SGD-SM 2.0 results of the location (38.375^∘ N, 117.125^∘ E) in 2005, 2010, 2015 and 2020.

5 Comparisons with SGD-SM 1.0

In this section, we compare the proposed SGD-SM 2.0 dataset with the previous SGD-SM 1.0 dataset, from the perspectives of reconstructing accuracy and time-series consistency. In contrast with SGD-SM 1.0, we fuse the global daily precipitation products into the reconstructing framework. In addition, the LSTM-CNN model is developed to fill the gaps and missing regions in SGD-SM 2.0 global daily soil moisture products. Detailed comparisons between the SGD-SM 1.0 and SGD-SM 2.0 datasets are displayed below.

5.1 Reconstructing accuracy

For ensuring the same time scope with SGD-SM 1.0, we choose the part of SGD-SM 2.0 from 2013 to 2019. The average evaluation indicators (R, RMSE, and MAE) of monthly-averaging, SGD-SM 1.0 and SGD-SM 2.0 datasets in selective 124 in situ sites are contrasted in Table 4.

Compared with monthly-averaging and SGD-SM 1.0 products, SGD-SM 2.0 products outperform on R (0.688), RMSE (0.094) and MAE (0.077). The main reason is that SGD-SM 1.0 ignores the sudden extreme weather conditions for 1 d. If a sudden precipitation occurs in 1 d, while there are no abnormalities before and after this day, SGD-SM 1.0 usually behaves with poor performance under this condition. Accordingly, SGD-SM 2.0 introduces the global daily precipitation products into the reconstructing framework. Through fusing auxiliary precipitation information, SGD-SM 2.0 products can consider the sudden extreme weather conditions for a single day in global daily soil moisture products. The comparisons validate the effectiveness of this point in Table 4.

Table 4 Comparisons between the SGD-SM 1.0 and SGD-SM 2.0 products (from 2013 to 2019) through selected 124 in situ sites. The best indexes are marked as bold format.

5.2 Time-series consistency

Except the reconstructing accuracy, time-series consistency is also significant for seamless products (Wang et al., 2021). As portrayed in Fig. 11a and b, we simultaneously depict time-series daily original soil moisture, SGD-SM 1.0/2.0, and precipitation results of the location (48.875^∘ N, 140.375^∘ E) in 2013, respectively. The blue dots refer to existing valid values in Fig. 11. The red dots stand for the SGD-SM 1.0/2.0 values in Fig. 11, which also represents the invalid gaps or missing soil moisture regions. The left vertical coordinate denotes the percent of soil moisture product in the original and SGD-SM 1.0/2.0 products. The right vertical coordinate refers to the daily precipitation value (unit: mm) by the IMERG precipitation products.

Compared with SGD-SM 1.0, SGD-SM 2.0 outperforms on time-series consistency in Fig. 11a and b. The reconstructed SGD-SM 2.0 dots behave more consecutively around their adjacent original SM dots than SGD-SM 1.0. A discrete problem in Fig. 11a exists in SGD-SM 1.0, to some degree. Benefiting from the data fusion of daily precipitation information, the proposed LSTM model can extract time-series features for filling the gaps and missing regions in daily soil moisture products. Therefore, SGD-SM 2.0 can be utilised effectively for global hydrology monitoring analysis at fine (daily) temporal resolution.

Figure 11 Time-series daily original soil moisture, SGD-SM 1.0/2.0, and precipitation results at location (48.875^∘ N, 140.375^∘ E) in 2013.

5.3 Uncertainty analysis

The uncertainty of SGD-SM 2.0 and the proposed model could be classified into three types: (1) the errors of original AMSR-E/WindSat/AMSR2 products; (2) the meteorological factors; (3) the generalisation of the proposed reconstructing model.

• The errors of original AMSR-E/WindSat/AMSR2 products. The proposed SGD-SM product is generated based on original AMSR-E/WindSat/AMSR2 products. Errors also exist in these passive soil moisture products (i.e. above 0.8 m³ m⁻³), due to the satellite sensor imaging and soil moisture retrieval algorithm. As shown in Table 1, the R, RMSE and MAE evaluation indexes of the original products are 0.679, 0.094 and 0.075, respectively. These errors are also inevitably transmitted into the generated SGD-SM 2.0 products. In other words, SGD-SM 2.0 absolutely trusts the initial satellite-based SM values without any hesitation.

• The meteorological factors. The proposed method relies on the temporal continuity and spatial consistency for daily soil moisture gap-filling. Nevertheless, if the unusual meteorology occurs in a single day, such as precipitation and snowfall, it may disturb the above-mentioned assumption and influence the reconstructing effects. This uncertainty can be noticed in time-series validation, especially for the rainy season. Although we fuse the daily precipitation products into the proposed model in SGD-SM 2.0, it still cannot adequately reflect the emergency meteorological factors such as brief precipitation.

• The generalisation of proposed reconstructing model. In this work, we train the proposed LSTM-CNN model through selecting complete soil moisture patches all over the world. In addition, the simulated masks are also chosen from the daily soil moisture products. However, differences between the training data and testing data still exist, such as land covering type and mask size. This uncertainty may disturb the generalisation of the proposed LSTM-CNN model for SGD-SM 2.0, to some degree.

6 Data availability

The proposed SGD-SM 2.0 dataset can be acquired from https://doi.org/10.5281/zenodo.6041561 (Zhang et al., 2022).

7 Conclusions

In this paper, we generate an improved seamless global daily soil moisture (SGD-SM 2.0) dataset from 2002 to 2022. Compared with the previous SGD-SM 1.0, the temporal range of SGD-SM 2.0 is extended to 20 years from 2002 to 2022. The SGD-SM 2.0 dataset fuses the global daily precipitation products with global daily soil moisture products. In addition, the SGD-SM 2.0 dataset develops an integrated LSTM-CNN model to fill the gaps and missing regions. In situ validation and time-series validation testify to the soil moisture time-series of SGD-SM 2.0 products (R: 0.672, RMSE: 0.096, MAE: 0.078). In contrast with SGD-SM 1.0, the time-series curves of the improved SGD-SM 2.0 products are consistency with the original daily time-series soil moisture values.

In our future work (SGD-SM 3.0), we will fuse multi-source data such as global land cover products and land surface temperature into the reconstructed framework. More spatio-temporal models will be exploited to generate the prospective products. In addition, we will introduce the outlier filtering strategy to exclude this initial soil moisture exception information (above 80 % volume. Identifying outliers by comparing the soil moisture data product with the porosity information from the global soil database SoilGrid will also be utilised in SGD-SM 3.0.