The AMSR-E, AMSR2 and WindSat global daily soil moisture products are utilised from 2002 to 2022. These three sensors are onboarded at the Aqua satellite, GCOM-W1 and Coriolis satellite, respectively (Nepal et al., 2021). The AMSR-E, AMSR2 and WindSat products are all passive sensors for soil moisture retrieving. The spatial resolution is a 0.25∘ grid (about 25 km) in all these products, as depicted in Fig. 1a–c. The retrieving model adopts the land parameter retrieval model (LPRM) for AMSR-E, WindSat and AMSR2 products (McColl et al., 2017). We select the descending orbit (nighttime) and 6.9 GHz band for all these soil moisture products. These datasets are all recorded in the GES DISC website. These three products provide the original information for using SGD-SM 2.0. The proposed reconstructing model acquires the gap masks and relies on the valid spatio-temporal soil moisture information from these three products to fill the gaps and missing regions.

The time-series range of the AMSR-E sensor starts from 19 June 2002 and ends on 4 October 2011 (Njoku et al., 2003; Shi et al., 2008). The time-series range of the WindSat sensor starts from 1 February 2003 and ends on 2 August 2012. The time-series range of the AMSR2 sensor starts from 3 July 2012 and continues to the current date (Zeng et al., 2020). In consideration of the low-coverage rate in the WindSat dataset, we only use WindSat global daily products from 5 October 2011 to 2 July 2012 for acquiring sequential daily products. These recorded AMSR-E, WindSat and AMSR2 global daily products are all employed as the initial input of the proposed LSTM-CNN model for generating SGD-SM 2.0 products. The daily coverage rate curves of these three global quantitative products are depicted in Fig. 2a–c, respectively.

Precipitation usually has a high correlation with soil moisture in the corresponding regions (Pellarin et al., 2009; Brocca et al., 2014a; Sun and Fu, 2021). Therefore, we fuse the precipitation products into the proposed SGD-SM 2.0 dataset to improve the reconstructing accuracy. The Integrated Multi-satellitE Retrievals for GPM (IMERG) global daily precipitation V6 products are employed for the years 2002–2022 (Massari et al., 2020). These precipitation products are derived from multiple precipitation-relevant satellite passive microwave sensors, as portrayed in Fig. 3a. The spatial resolution is denoted as a 0.1∘ grid (about 10 km) in IMERG level 3 global daily final precipitation products. To keep the uniformity with soil moisture products, the spatial downsampling operation is carried out for the original IMERG precipitation products from 0.1 to 0.25∘. We then normalise these precipitation values via linear transformation for use in the reconstructing model. These precipitation products were all downloaded from GES DISC (Brocca et al., 2019; Berg et al., 2021; Škrk et al., 2021).

In situ soil moisture sites are significant for testifying to the satellite-based products (Brocca et al., 2014b; Gruber et al., 2020). These sites provide high-precision surface soil moisture values. Relying on in situ data, the quantitative indexes could be derived for the proposed SGD-SM 2.0 dataset. The ISMN unites global in situ surface data, which have been widely applied for hydrology and soil moisture validation (Dorigo et al., 2011, 2013, 2021; Wigneron et al., 2013). We select 124 stations from the ISMN from 2002 to 2022 and match them with corresponding soil moisture products in SGD-SM 2.0 (Zhang et al., 2020). The selected criteria include three points:

The in situ soil moisture sites are downloadable through the given website.

As shown in Fig. 4, the original global daily soil moisture product on date T (AMSR-E, WindSat or AMSR2) and its corresponding global daily precipitation product are utilised as the input data of the proposed framework. Firstly, the precipitation data on date T are transformed as the vector value PT through a full-connected (FC in Fig. 4) layer. We employ the partial convolutional neural network (Partial CNN in Fig. 4) to extract the spatial feature of soil moisture product on date T. Different from the common CNN (Yuan et al., 2019), partial CNN can effectively acquire the spatial information within valid regions and eliminate the invalid information within gaps or soil moisture missing regions (Zhang et al., 2018a). We applied a partial CNN for generating the SGD-SM 1.0 dataset. Due to its effectiveness on incomplete soil moisture products, the partial CNN is also used in this work for generating the SGD-SM 2.0 dataset. The formula of a partial CNN in this work is determined as follows: 1S′(m,n)=W⊤(S(m,n)⊗M(m,n))1(m,n)1M(m,n)1+b,M(m,n)1≠00,otherwise, where M denotes the mask of its corresponding soil moisture product S; 0 and 1 refer to the invalid and valid points in the mask M. The trainable weighted and offset arguments in the partial CNN are denoted by W and b, respectively; and ⊗ represents the dot product operation to exclude the invalid information in gaps or missing regions through mask data. Subsequently, the current mask M needs to be regenerated under the below paradigm: if the partial convolution can generate at least one valid value of the output result, we need to mark this location as a valid value in the new masks. The regenerated mask formula is defined as follows (Zhang et al., 2020): 2M′(m,n)=L(m,n),M(m,n)1≠00,other, where L(m,n) stands for Earth land in position (m, n). It should be noted that the Earth land mask includes six continents and neglects all regions of Antarctica and most regions of Greenland. The main reason is that these omitted regions are perennially covered with snowy or frozen land (Zhao et al., 2021).

After four partial CNN layers in Fig. 4, an FC layer is also acted on in the feature maps of the soil moisture product, with the result of vector value ST: 3ST=FC(S′4)s.t.PT=FC(PT).

Then, the two vectors, ST and PT, of soil moisture and precipitation products are simultaneously imported into the LSTM model in Fig. 4. The architecture of the LSTM model within the proposed framework is displayed in Fig. 5.

As depicted in Fig. 5, soil moisture information ST, corresponding precipitation information PT, previous long-term memory information CT-1, and previous short-term memory information hT-1 are simultaneously imported into the LSTM model. The output values in the LSTM are the regenerated soil moisture information S′T, current long-term memory information CT, and current short-term memory information hT. It should be noted that current LSTM information on date T is the previous LSTM information on next date T+1, respectively. For memory information h0 and C0, these vectors are initialised with zero elements. The LSTM is composed of three gates, i.e. oblivious gate, input gate and output gate to control the memory information state (Zhang et al., 2021b).

Oblivious gate. This gate determines which information needs to be discarded in the short-term memory state. It is carried out by the sigmoid unit σ between the soil moisture information ST and previous long-term memory information CT-1:4fT=σ(Wf⋅[hT-1,ST]+bf),where the sigmoid unit σ is defined below:5σ(a)=11+e-a.

In the sigmoid unit σ, 0 means fail and 1 means pass for current information. Through checking the soil moisture information ST and previous long-term memory information CT-1, it generates a vector between 0 and 1. This variable determines which information is retained or discarded in the short-term memory state.

Besides, we need to output current short-term memory information hT for the next date T+1 as follows: 11hT=tanh⁡(oT⊗CT).

Later, the regenerated soil moisture information S′T is transformed by an FC layer in the right of Fig. 4. Thereafter, four partial CNN layers are performed to generate the final SGD-SM 2.0 product on date T. Through the consecutive time-series strategy, we recursively reconstruct the daily soil moisture products in SGD-SM 2.0.

To generate the reliable and high-precision SGD-SM 2.0 dataset, the way in which to train and optimise the proposed LSTM-CNN model is extremely crucial in this work. The training stage needs huge numbers of sample labels to optimise the trainable parameters in the proposed partial CNN and LSTM in Figs. 4 and 5, respectively. The sample labels adopt a patch-selecting strategy. We select sequential time-series daily soil moisture patches with k=7 in the reconstructing framework. The spatial sizes of these 7 d soil moisture patches are all set as 40×40. These time-series 7 d soil moisture patches are all complete, without gaps or data missing regions from the original soil moisture products. Then, we randomly select 30 000 mask patches with the spatial size of 40×40. Each soil moisture patch is simulated with missing regions via these mask patches. In this way, we acquire 30 000 training samples from the original 2002–2022 soil moisture products. Each training sample includes four variables: the simulated 7 d soil moisture patches, the complete 7 d soil moisture patches, the corresponding mask patches, and the corresponding precipitation patches. These variables are simultaneously imported into the LSTM-CNN reconstructing model, as shown in Fig. 4.

For the partial CNN in the proposed framework, we set the convolutional filter size as 3×3 in all the partial CNN layers (Xiao et al., 2022a). The last partial CNN layer outputs just one feature map and the other partial CNN layers output 64 feature maps (Xiao et al., 2022b). The rectified linear unit (ReLU) is utilised after each partial CNN layer. For the LSTM model in the proposed framework, we set the dimension of long and short-term memory vectors CT and hT as 2048.

For the network optimisation, we adopt the same strategy with the global–local function (Zhang et al., 2021a) in SGD-SM 1.0. The global soil moisture uniformity and local soil moisture heterogeneity are both taken into consideration in the proposed LSTM-CNN reconstructing model. Different from SGD-SM 1.0, we simultaneously fill the gaps and missing regions in the time-series 7 d soil moisture patches. Detailed definitions of the global–local function are determined as follows: 12ξ(W,b,Wf,i,C,S′,bf,i,C,S′)=∑T=1k(1-MT)⊗(STrec-STori)22+ML⊗(STrec-STori)22, where ML represents the global land mask (including six continents and neglecting all regions of Antarctica and most regions of Greenland); α stands for the balancing parameter to equilibrate the local loss and global loss (Zhang et al., 2020). Empirically, this ratio is fixed as 0.1 in the training and optimisation stage.

In terms of the hyperparameters and operations of the proposed framework, related explanations are listed below. The batch size of the LSTM-CNN reconstructing model is set as 128 (Zhang et al., 2018a). The whole epoch number is confirmed, determined as 500 (1 epoch denotes that all the samples in the training set have been utilised for the neural network optimisation at one time). The inceptive learning rate is started at 0.005 (Zhang et al., 2018b). It gradually decreases through multiplying a damping factor (equal to 0.5) every 100 epochs (Zhang et al., 2019). On software configuration, the LSTM-CNN model is carried out on the PyTorch 1.8.1 framework. We use Python 3.7 language, PyCharm platform and the Windows 10 environment to generate seamless global daily soil moisture products. On hardware configuration, we employ a NVIDIA Titan X (Pascal) GPU, Inter E5-2609v3 CPU, and 16 GB DDR4 RAM to execute the proposed LSTM-CNN model.

As shown in Figs. 6 and 7, the original SM and SGD-SM 2.0 results are given on 10, 20 and 30 September 2002 and on 10, 20 and 30 June 2020, respectively.

For comparison purposes, the left lines are the original global daily products and the right lines are the reconstructed SGD-SM 2.0 products in Figs. 6 and 7. It should be noted that we neglect all regions in Antarctica and most regions in Greenland because of the perpetual frozen soil. Clearly, gaps and missing regions are filled through the proposed framework in Sect. 3.

From the spatial perspective, the proposed SGD-SM 2.0 dataset performs both global soil moisture uniformity and local soil moisture heterogeneity in Figs. 6 and 7. It ensures the spatial consistency, especially for the gap regions with the adjoining soil moisture regions. Beyond that, the reconstructed regions in SGD-SM 2.0 do not reflect distinct patch or border effects. This also testifies to the powerful ability of the partial CNN in the proposed framework, which can effectively exclude the invalid information in gaps or missing soil moisture regions.

From the temporal perspective, the proposed SGD-SM 2.0 dataset utilises the complementary and sequential time-series soil moisture information. By fusing global daily precipitation products, SGD-SM 2.0 can consider the sporadic extreme weather conditions for a single day. In addition, by means of the LSTM model, the consistent temporal information can be recovered and preserved in Figs. 6 and 7.

In situ validation is the most reliable method to measure the accuracy and availability of the proposed SGD-SM 2.0 dataset (Walker et al., 2004; Draper et al., 2009; Zeng et al., 2015b). In this work, we choose 124 in situ surface (0–5 cm depth) soil moisture sites from the ISMN, as shown in Fig. 3b. The selected in situ values are limited from 2002 to 2022. We match the hourly in situ values with the descending products. In consideration of validation reliability, we choose the two neighbouring in situ values that correspond with the observation time of soil moisture products. We then average them as the ground-truth data.

As portrayed in Fig. 8, the scatters of six in situ soil moisture sites (marked as blue circles in Fig. 3b: 42.537∘ N, 72.171∘ W; 0.282∘ N, 36.866∘ E; 48.141∘ N, 15.171∘ E; 14.159∘ S, 131.388∘ E; 21.617∘ S, 47.632∘ W; 31.369∘ N, 91.899∘ E) are displayed to demonstrate the reconstructing accuracy of SGD-SM 2.0. The horizontal coordinate refers to in situ data. Accordingly, the vertical coordinate denotes reconstructing data in gaps or missing soil moisture regions. The time range is limited from 2002 to 2022. The R indicators of these sites are varied from 0.658 to 0.769 in Fig. 8a–f. The RMSE and MAE indicators of these sites are varied from 0.023 to 0.144 and from 0.021 to 0.128, respectively in Fig. 8a–f.

In all 124 selected in situ sites, Table 1 compares the original products with the SGD-SM 2.0 products. The average evaluation indicators (R, RMSE, and MAE) of the original SM (SGD-SM 2.0) products are 0.679 (0.672), 0.094 (0.096) and 0.075 (0.078), respectively. Generally, the precision of SGD-SM 2.0 products performs similar with incipient products. The diversities of those indicators are few between the original and reconstructed SGD-SM 2.0 products in Table 1. To a certain extent, in situ validation testifies to the reconstructed accuracy and validity of the SGD-SM 2.0 products. These 124 selected soil moisture stations from the ISMN (2002–2022) are shown in Table 2 for validating SGD-SM 2.0. Besides, basic information about the representative in situ soil moisture sites (taking partial sites as examples, including COSMOS, SD-DEM, SMOS-CBR, SCAN, PBO, USCRN and OZNET networks) is listed in Table 3. As the example, it includes the name of the station, country, longitude/latitude, main land use, lattice water, and soil organic carbon.

Long-term daily soil moisture products usually reflect typical time-series continuity (Wang et al., 2022; Seneviratne et al., 2010). Therefore, we can utilise this characteristic to validate the reliability of SGD-SM 2.0 products. As listed in Figs. 9 and 10, 2 time-series daily original/SGD-SM 2.0 results of 2003–2018 and 2005–2020, are given in the location (10.125∘ S, 42.625∘ W) and the location (38.375∘ N, 117.125∘ E), respectively. The blue dots refer to the existing valid values in Figs. 9 and 10. The red dots stand for the SGD-SM 2.0 values in Figs. 9 and 10, which also represents the invalid gaps or missing soil moisture regions. The vertical coordinate denotes the percent of soil moisture product in original and SGD-SM 2.0 products. The horizontal coordinate denotes the annual date number between 2003 and 2020.

As depicted in Figs. 9a–d and 10a–d, a majority of the reconstructed SGD-SM 2.0 values (in invalid gaps or missing soil moisture regions) can distinctly embody the time-series continuity. In the two locations of different years, original soil moisture values and corresponding adjacent SGD-SM 2.0 values perform fore-and-aft consistency. If current valid soil moisture values behave high or low, their neighbouring SGD-SM 2.0 values also accord with them in Figs. 9 and 10. This time-series validation manifests the reliability of the proposed framework and validity of our improved SGD-SM 2.0 products. Generally, the proposed SGD-SM 2.0 products are able to ensure the time-series continuity in daily temporal resolution. This point is greatly important for reconstructing long-term products. Benefiting from the utilisation of temporal information, the proposed LSTM model can extract and transmit time-series features for filling the gaps and missing data regions in daily soil moisture products. Therefore, SGD-SM 2.0 can be effectively applied for global hydrology monitoring analysis at fine-temporal scale, rather than the traditional monthly or yearly averaging operation. The former preserves the original daily temporal resolution, while the latter sacrifices this daily temporal resolution. This validation exactly demonstrates the above significance of the proposed SGD-SM 2.0 dataset.

For ensuring the same time scope with SGD-SM 1.0, we choose the part of SGD-SM 2.0 from 2013 to 2019. The average evaluation indicators (R, RMSE, and MAE) of monthly-averaging, SGD-SM 1.0 and SGD-SM 2.0 datasets in selective 124 in situ sites are contrasted in Table 4.

Compared with monthly-averaging and SGD-SM 1.0 products, SGD-SM 2.0 products outperform on R (0.688), RMSE (0.094) and MAE (0.077). The main reason is that SGD-SM 1.0 ignores the sudden extreme weather conditions for 1 d. If a sudden precipitation occurs in 1 d, while there are no abnormalities before and after this day, SGD-SM 1.0 usually behaves with poor performance under this condition. Accordingly, SGD-SM 2.0 introduces the global daily precipitation products into the reconstructing framework. Through fusing auxiliary precipitation information, SGD-SM 2.0 products can consider the sudden extreme weather conditions for a single day in global daily soil moisture products. The comparisons validate the effectiveness of this point in Table 4.

Except the reconstructing accuracy, time-series consistency is also significant for seamless products (Wang et al., 2021). As portrayed in Fig. 11a and b, we simultaneously depict time-series daily original soil moisture, SGD-SM 1.0/2.0, and precipitation results of the location (48.875∘ N, 140.375∘ E) in 2013, respectively. The blue dots refer to existing valid values in Fig. 11. The red dots stand for the SGD-SM 1.0/2.0 values in Fig. 11, which also represents the invalid gaps or missing soil moisture regions. The left vertical coordinate denotes the percent of soil moisture product in the original and SGD-SM 1.0/2.0 products. The right vertical coordinate refers to the daily precipitation value (unit: mm) by the IMERG precipitation products.

Compared with SGD-SM 1.0, SGD-SM 2.0 outperforms on time-series consistency in Fig. 11a and b. The reconstructed SGD-SM 2.0 dots behave more consecutively around their adjacent original SM dots than SGD-SM 1.0. A discrete problem in Fig. 11a exists in SGD-SM 1.0, to some degree. Benefiting from the data fusion of daily precipitation information, the proposed LSTM model can extract time-series features for filling the gaps and missing regions in daily soil moisture products. Therefore, SGD-SM 2.0 can be utilised effectively for global hydrology monitoring analysis at fine (daily) temporal resolution.

The uncertainty of SGD-SM 2.0 and the proposed model could be classified into three types: (1) the errors of original AMSR-E/WindSat/AMSR2 products; (2) the meteorological factors; (3) the generalisation of the proposed reconstructing model.

The errors of original AMSR-E/WindSat/AMSR2 products. The proposed SGD-SM product is generated based on original AMSR-E/WindSat/AMSR2 products. Errors also exist in these passive soil moisture products (i.e. above 0.8 m3 m-3), due to the satellite sensor imaging and soil moisture retrieval algorithm. As shown in Table 1, the R, RMSE and MAE evaluation indexes of the original products are 0.679, 0.094 and 0.075, respectively. These errors are also inevitably transmitted into the generated SGD-SM 2.0 products. In other words, SGD-SM 2.0 absolutely trusts the initial satellite-based SM values without any hesitation.