Generating seamless global daily AMSR2 soil moisture (SGD-SM) long-term products for the years 2013–2019

High-quality and long-term soil moisture products are significant for hydrologic monitoring and agricultural management. However, the acquired daily Advanced Microwave Scanning Radiometer 2 (AMSR2) soil moisture products are incomplete in global land (just about 30 %–80 % coverage ratio), due to the satellite orbit coverage and the limitations of soil moisture retrieval algorithms. To solve this inevitable problem, we develop a novel spatio-temporal partial convolutional neural network (CNN) for AMSR2 soil moisture product gap-filling. Through the proposed framework, we generate the seamless daily global (SGD) AMSR2 long-term soil moisture products from 2013 to 2019. To further validate the effectiveness of these products, three verification methods are used as follows: (1) in situ validation, (2) time-series validation, and (3) simulated missing-region validation. Results show that the seamless global daily soil moisture products have reliable cooperativity with the selected in situ values. The evaluation indexes of the reconstructed (original) dataset are a correlation coefficient (R) of 0.685 (0.689), root-mean-squared error (RMSE) of 0.097 (0.093), and mean absolute error (MAE) of 0.079 (0.077). The temporal consistency of the reconstructed daily soil moisture products is ensured with the original time-series distribution of valid values. The spatial continuity of the reconstructed regions is in accordance with the spatial information (R: 0.963–0.974, RMSE: 0.065–0.073, and MAE: 0.044–0.052). This dataset can be downloaded at https://doi.org/10.5281/zenodo.4417458 (Zhang et al., 2021).

1 Introduction

Surface soil moisture is a crucial Earth land characteristic in describing the hydrologic cycle system (Wigneron et al., 2003; Lievens et al., 2015). It can be applied for monitoring droughts and floods in agriculture (Samaniego et al., 2018) and geologic hazards (Long et al., 2014). To obtain the global and high-frequency soil moisture products, many active or passive satellite sensors have been launched, such as the Advanced Microwave Scanning Radiometer for Earth Observing System (AMSR-E), Advanced Microwave Scanning Radiometer 2 (AMSR2), and Soil Moisture Active and Passive (SMAP), Soil Moisture and Ocean Salinity (SMOS) among others (McColl et al., 2017; Ma et al., 2019). Nevertheless, the acquired daily soil moisture products are always incomplete in global land (see Fig. 1a, about 30 %–80 % missing ratio in AMSR2), because of the satellite orbit coverage and the limitations of soil moisture retrieval algorithms (Cho et al., 2017; Long et al., 2019). The invalid land regions refer to areas with a gap or missing information. Especially in the regions close to the Equator, or in the permafrost region, the degree of missing soil moisture data is more serious (Zeng et al., 2015; Santi et al., 2018). This phenomenon greatly disturbs subsequent soil moisture applications, especially for the consecutive daily temporal analysis and global spatial distribution comparisons (Colliander et al., 2017; Liu et al., 2019).

To reduce this negative effect, most existing works employed the strategy of multi-temporal soil moisture data selection, multi-temporal soil moisture data averaging, or multi-temporal soil moisture data synthesis. Detailed descriptions and analyses of these three strategies (Al Bitar et al., 2017) are presented as follows.

1. Multi-temporal soil moisture data selection. The criterion of this strategy denotes selection of the highest coverage regions on a single date from multi-temporal soil moisture products (Wang and Qu, 2009). However, this assumption can only deal with local regions and is not applicable for global regions. The main reason is that almost all the global daily soil moisture products suffer from the defect of missing satellite orbit coverage and failure of the retrieval algorithm. Multi-temporal soil moisture data selection strategy greatly reduces the data utilisation and is not qualified for dense time-series analysis at daily temporal resolution (Liu et al., 2020; Purdy et al., 2018).

2. Multi-temporal soil moisture data averaging. This strategy is commonly used for most soil moisture studies or applications. The incomplete soil moisture products are overall averaged as the monthly, quarterly, and yearly results to generate the complete products (Jalilvand et al., 2019). For most applications and spatial analysis, this operation can effectively improve the spatial soil moisture coverage (Zhao et al., 2020). However, it distinctly sacrifices the high-frequency temporal resolution as low-frequency temporal resolution, which also severely reduces the data utilisation. In addition, it ignores the unique spatial distribution of a single day and loses the dense time-series changing information. In other words, the monthly, quarterly, and yearly soil moisture data averaging operations damage the initial information on both spatial and temporal dimensions.

3. Multi-temporal soil moisture data synthesis. Different from soil moisture data selection and averaging, this strategy employs the time-series daily soil moisture data and selects the valid observed value from corresponding time-series pixels. This strategy can produce synthesis results through valid single points, while it ignores the local spatial correlation and contains discontinuous and inconsistent effects in local regions. In addition, it also sacrifices high temporal resolution just as the multi-temporal data averaging strategy does (Peng et al., 2017; Sun et al., 2020).

To overcome the above-mentioned limitations, some missing-value reconstruction methods have been developed in particular for multi-temporal images, thick cloud removal, and deadline gap-filling (Q. Zhang et al., 2020a). For example, Zhu et al. (2011) proposed the multi-temporal neighbouring-homologous-value padding method for thick cloud removal. Chen et al. (2011) presented an effective interpolating algorithm for recovering the invalid regions in Landsat images. Zhang et al. (2018a) built an integrative spatio-temporal spectral network for missing data reconstruction in multiple tasks.

In terms of the soil moisture product gap-filling, several methods have also been proposed to address this issue. Wang et al. (2012) presented a penalised least-square regression-based approach for global satellite soil moisture gap-filling observation. Fang et al. (2017) introduced a long short‐term-memory network to generate a spatially complete overlay SMAP in the US. Long et al. (2019) fused multi-resolution soil moisture products, which can produce daily fine-resolution data in local regions. Llamas et al. (2020) used geostatistical techniques and a multiple regression strategy to get spatially complete results of satellite-derived products. Overall, there are few works for soil moisture reconstruction on global and daily scales.

In spatial dimension, the invalid land areas and adjacent valid land areas show spatial consistency and spatial correlation for daily soil moisture products (Long et al., 2020). In temporal dimensions, the daily time-series changing curve of the same point natively appears with continuous and smooth peculiarities (Chan et al., 2018). Overall, these methods can effectively fill the gaps of soil moisture products. However, these methods cannot simultaneously take both spatial and temporal information into consideration. In addition, the daily soil moisture products on a global scale have not been exploited up to now.

Therefore, how about simultaneously extracting both spatial and temporal features for seamless global daily soil moisture product gap-filling? Recently, deep learning has gradually revealed the potential for remote sensing products processing (Chen et al., 2021). In consideration of the powerful feature expression ability via deep learning, can we utilise spatio-temporal information to generate long-term soil moisture products?

From these perspectives, a novel spatio-temporal deep learning framework is proposed for global daily AMSR2 soil moisture products gap-filling. By means of the proposed method, we can effectively break through the above-mentioned limitations. And finally, this work generates the seamless global daily AMSR2 soil moisture long-term products from 2013 to 2019. The main innovations are summarised below.

1. We develop a deep 3D partial reconstruction model, which can take both the spatial and temporal information into consideration. Aiming at the invalid or coastline region boundary, the 3D partial CNN and global–local loss function are presented for better extracting the valid region features and ignoring the invalid regions through both soil moisture data and mask information.

2. A seamless global daily (SGD) AMSR2 soil moisture long-term (2013–2019) dataset is generated through the proposed model. The dataset includes the original and reconstructed soil moisture data. These SGD products could be directly downloaded at https://doi.org/ /10.5281/zenodo.4417458 (Zhang et al., 2021).

3. Three verification strategies are employed to evidence the precision of our SGD soil moisture dataset as follows: in situ validation, time-series validation, and simulated missing-region validation. Evaluation indexes demonstrate that the seamless global daily AMSR2 soil moisture dataset shows high accuracy, reliability, and robustness.

The schema of this work is listed below. Section 2 describes the ASMR2 soil moisture products and in situ soil moisture network data. Section 3 presents the methodology for generating the seamless global daily AMSR2 soil moisture products. Section 4 gives the experimental results and related validation results. The comparisons between the time-series averaging method and proposed method are discussed in Sect. 5. Finally, Sect. 7 lists the conclusions of this study.

2 Data description

2.1 AMSR2 soil moisture products

In consideration of the global coverage, temporal resolution, and current availability, we select AMSR2 soil moisture products as the focus of this study. This sensor was on board on the Global Change Observation Mission 1-Water (GCOM-W1) satellite, launched in May 2012 (Kim et al., 2015). The released datasets include three passive microwave band frequencies: 6.9 GHz (C1 band), 7.3 GHz (C2 band, new frequency compared with AMSR-E), and 10.7 GHz (X band). It can observe the global land two times within a day (Wu et al., 2016): ascending (day-time) and descending (night-time, about 00:00–01:00 of the local time) orbits. The primary spatial resolution of this dataset denotes 0.25^∘ global grids. And the AMSR2 soil moisture retrieval algorithms include the Land Parameter Retrieval Model (LPRM) and Japan Aerospace Exploration Agency (JAXA) (Du et al., 2017; Kim et al., 2018). The error of soil moisture for each frequency was also given in AMSR2 products.

In our study, we choose LPRM AMSR2 descending level 3 (L3) global daily 0.25^∘ soil moisture products as the study data. To avoid introducing additional error and uncertainty, we did not carry out the downscaling operation of the generated SGD-SM products. This dataset was obtained from https://hydro1.gesdisc.eosdis.nasa.gov/ (last access: 18 June 2020). For instance, the original AMSR2 0.25^∘ soil moisture data in 2 April 2019 are displayed in Fig. 1a. Due to the satellite orbit coverage and limitations of soil moisture retrieving algorithms in tundra areas (Muzalevskiy et al., 2020), the acquired AMSR2 daily soil moisture products are always incomplete in global land (about 30 %–80 % invalid ratio, excluding Antarctica and most of Greenland), as shown in Fig. 1a. The daily global land coverage ratio of AMSR2 soil moisture data in 2019 is listed in Fig. 2. Distinctly, the global land coverage ratio is low in wintertime and high in summertime. The mean global land coverage ratio in 2019 is just about 56.5 % in AMSR2 soil moisture daily products. Apparently, these incomplete soil moisture data cannot be directly applied for subsequent spatial and time-series analysis, as mentioned in the previous Sect. 1.

Figure 1 AMSR2 soil moisture product and selected in situ soil moisture sites.

Figure 2 The daily global land coverage ratio of AMSR2 soil moisture products in 2019.

2.2 International Soil Moisture Network in situ data

The International Soil Moisture Network (ISMN) was established from 2009 to now (Dorigo et al., 2011), providing the correction and validation schemes for remote sensing satellite-based soil moisture retrieval. ISMN includes the globally distributed in situ soil moisture sites supported by the earth observation of the European Space Agency (ESA) and many voluntary contributions of researchers and organisations from all over the world (Dorigo et al., 2012, 2013).

The ISMN in situ surface soil moisture values could be acquired through https://ismn.geo.tuwien.ac.at (last access: 21 July 2020). In our experiments, we selected a portion of in situ soil moisture sites of ISMN as ground truth values (Zhang et al., 2017), to testify the precision and credibility of the reconstructed datasets in Sect. 4.2. The spatial distribution of the used in situ sites is depicted in Fig. 1b. It should be noted that the time range is restrained from 1 January 2013 to 31 December 2019. Then the daily soil moisture values are matched with the in situ sites in the same location. Two neighbouring in situ hourly values are averaged as the ultimate result of the current date (Dong et al., 2020).

3 Methodology

The flow chart of the presented framework is depicted in Fig. 3. The overall structure could be divided into two stages: the training procedure and testing procedure. Firstly, we designate the processing daily soil moisture data on date T and simultaneously select its adjacent time-series data before and after 4 d (date T−4 to T+4). The corresponding land masks of these daily soil moisture data are generated through the invalid pixel marking.

In the training procedure, these spatio-temporal soil moisture data and land mask patch groups are imported as the training data of the presented spatio-temporal 3-D reconstruction model through patch selection and mask simulation. The convergence condition denotes that the loss of the proposed model gradually decreases and finally remains smooth in the training procedure through back-propagation (BP) in Fig. 3. Then in the testing procedure, seamless global daily reconstruction soil moisture data are outputted through the convergent model. Subsequently, the next daily processing soil moisture data are designated and the above-mentioned steps are repeated, until all the daily data are serially reconstructed in order. Details of the reconstruction model and network are described below.

Figure 3 Flow chart of the presented framework.

3.1 Spatio-temporal 3-D reconstruction model

The spatio-temporal soil moisture reconstruction model is displayed in Fig. 4. After assigning the original soil moisture data on date T, time-series soil moisture data and corresponding masks on dates T−4 to T+4 are simultaneously imported as the 3D-tensor inputs of the presented deep reconstruction model in Fig. 4. In the spatial dimension, missing and non-missing areas contain spatial consistency in daily soil moisture data. In the temporal dimension, the daily time-series changing curve of the same point natively appears with the continuous and smooth peculiarities. Therefore, the 3D CNN is employed to process the spatio-temporal soil moisture data in this model. In this way, we can jointly utilise both spatial and temporal information of these time-series soil moisture products. Further, it can better exploit the deep spatio-temporal feature for data reconstruction and model optimisation. The structure and details are depicted in Fig. 4.

This network includes 11 layers (3D partial CNN unit and ReLU (rectified linear unit)) in Fig. 4. The size of 3D filters is all set as $\mathrm{3}\times \mathrm{3}\times \mathrm{3}$. The number of feature maps before 10 layers is fixed as 90, and the channel of the feature map in the final layer is exported as 1. It should be noted that after finishing each partial 3D-CNN layer, we must update all the new masks for the next layer. The mask updating operation is defined in Sect. 3.2. In terms of the model training and optimisation, three steps, patch selecting, mask simulating, and back propagation, are performed in Sect. 3.3. Detailed technique descriptions of the network implementation are provided in the Supplement. For network optimisation, we take the global loss and local loss into consideration. As described in Fig. 3, this deep reconstruction model needs to learn with large training label samples, before the testing procedure for outputting global seamless daily soil moisture products. The global land mask and the mask on the current date T are also employed for the global loss and local loss in Fig. 4. Descriptions of partial 3D-CNN and model optimisation are demonstrated in Sect. 3.2 and 3.3, respectively.

Figure 4 Spatio-temporal soil moisture 3D reconstruction model.

3.2 Partial convolutional neural network

Deep convolution neural network have been widely applied for nature image reconstruction (Liu et al., 2018a; Yeh et al., 2017; Liu et al., 2019) and satellite imagery recovery (Yuan et al., 2019; Zhang et al., 2019; Q. Zhang et al., 2020b). Nevertheless, it should be highlighted that valid and invalid pixels simultaneously exist, especially around coastal regions and gap regions (Pathak et al., 2016). The common CNN ignores the location information of invalid or valid pixels in soil moisture data, which cannot eliminate the invalid information (Liu et al., 2018b). Therefore, to solve this negative effect, we have developed the partial 3D-CNN to ignore the invalid information in the proposed reconstruction model. Before introducing the partial convolution, the operation of common convolution can be defined as below:

$$\begin{array}{}\text{(1)}& x={\mathbf{W}}^T\mathbf{X}+b,\end{array}$$

where X denotes the input tensor data. W and b are the weight and bias parameters, respectively. Different from the common convolution, the mask information M of the corresponding soil moisture data is introduced into the partial convolution:

$$\begin{array}{}\text{(2)}& \begin{array}{rl}& x^{\prime }=\\ & \left\{\begin{array}{cc}{\mathbf{W}}^T({\mathbf{X}}_{(w,h,t)}\odot {\mathbf{M}}_{(w,h,t)})\frac{{∥{\mathbf{1}}_{(w,h,t)}∥}_{\mathrm{1}}}{{∥{\mathbf{M}}_{(w,h,t)}∥}_{\mathrm{1}}}+b,& {∥{\mathbf{M}}_{(w,h,t)}∥}_{\mathrm{1}}\ne \mathrm{0}\\ \mathrm{0},& \text{otherwise}\end{array}\right.,\end{array}\end{array}$$

where ⊙ stands for the pixel-wise multiplication. w, h, and t refer to the width, height, and temporal number of the input data, respectively. 1 denotes the identical dimension tensor with mask M, whose elements are all a value of 1. Obviously, the partial convolutional output x^′ is only decided by the valid soil moisture pixels of input X, rather than the invalid soil moisture pixels. Through the mask M, we can effectively exclude the interference information of invalid soil moisture pixels such as marine regions and gap regions. Then the scaling divisor in Eq. (2) further adjusts for the variational number of valid soil moisture pixels.

After finishing each partial convolution layer, all the masks need to be updated through the following rule: if the partial convolution can generate at least one valid value of the output result, then we mark this location as a valid value in the new masks. This updating operation is demonstrated as below:

$$\begin{array}{}\text{(3)}& {m^{\prime }}_{(w,h,t)}=\left\{\begin{array}{cc}{\mathrm{Land}}_{(w,h)}\cdot \mathrm{1},& {∥{\mathbf{M}}_{(w,h,t)}∥}_{\mathrm{1}}\ne \mathrm{0}\\ \mathrm{0},& \text{otherwise}\end{array}\right.,\end{array}$$

where Land_(w,h) is the global land mask at the location (w,h) of the global soil moisture product. This global land mask covers six continents and excludes Antarctica and most of Greenland.

3.3 Model training and optimisation

As shown in Fig. 3, the training procedure needs to generate large numbers of training samples for learning the proposed spatio-temporal 3-D reconstruction model in Fig. 4. Different from the testing procedure, the training procedure additionally contains the patch selection, mask simulation, and back propagation (BP) steps. These three steps are significant for model training and optimisation. The purpose of the patch selection and mask simulation steps in Fig. 3 is to establish the label (complete) data (incomplete) training samples in the deep learning framework. The significance of the BP step in Fig. 3 is to optimise the reconstruction network in Fig. 4 and acquire the loss convergence model for testing use.

In the patch selection step, we traverse the global regions on date T to select the complete soil moisture patch label, whose local land regions are undamaged. It should be noted the other incomplete patches on date T are excluded because they cannot participate in the supervised learning. The corresponding time-series soil moisture patches of this selected patch between date T−4 and T+4 are set as the spatio-temporal 3D data patch groups. And their corresponding masks between dates T−4 and T+4 are set as the spatio-temporal 3D mask patch groups. After traversing the original global daily AMSR2 soil moisture products from 2013 to 2019, we finally establish the spatio-temporal data and mask patch groups with 276 488 patches. The soil moisture patch size is fixed as 40×40 for patch selection.

In the mask simulation step, 10 000 patch masks of the size 40×40 are chosen from the global AMSR2 soil moisture masks from 2013 to 2019. The missing ratio range of these masks is set as [0.3, 0.7]. Then these patch masks are randomly selected for label patch use within the spatio-temporal data and mask patch groups. The complete patch on date T (label) is simulated as the incomplete patch (data) through the above mask. And the original corresponding mask of this patch also needs to be replaced. After traversing and building the label data 3D spatio-temporal patch groups, this dataset is set as the training sample for use of the reconstruction network in Fig. 3.

In the back propagation step, we need a loss function to iteratively optimise the learning parameters of the deep reconstruction network. This operation follows the chain rule in model optimising. The Euclidean loss function is employed in most data reconstruction or regression issues based on deep learning, such as satellite product downscaling (Fang et al., 2020) and retrieval (Lee et al., 2019). Nevertheless, Euclidean loss function only pays attention to the holistic information bias for network optimisation. It ignores the soil moisture particularity of the local areas, especially in local coastal, mountain, and hinterland regions. However, this particularity is extremely significant for invalid region gap-filling, because of the spatial heterogeneity in soil moisture products. Therefore, to take both the global consistency and local soil moisture particularity into consideration, the global land mask and current mask on date T are both employed after the final layer as shown in Fig. 3. Further, the reconstruction network presents the local and global two-norm loss as below:

$$\begin{array}{}\text{(4)}& {\displaystyle }{\mathit{\zeta }}_{\mathrm{local}}={∥(\mathrm{1}-{\mathbf{M}}_T)\odot ({\mathbf{SM}}_{\mathrm{rec}}-{\mathbf{SM}}_{\mathrm{ori}})∥}_{\mathrm{2}}^{\mathrm{2}},\text{(5)}& {\displaystyle }{\mathit{\zeta }}_{\mathrm{global}}={∥{\mathbf{M}}_{\mathrm{G}}\odot ({\mathbf{SM}}_{\mathrm{rec}}-{\mathbf{SM}}_{\mathrm{ori}})∥}_{\mathrm{2}}^{\mathrm{2}},\end{array}$$

where M_T stands for the current mask patch on date T. M_G represents the corresponding global land mask patch. SM_rec and SM_ori denote the reconstructed soil moisture patch and original seamless soil moisture patch, respectively. The unified loss function of the reconstruction network combines ζ_local and ζ_global as below:

$$\begin{array}{}\text{(6)}& \mathit{\zeta }\left(\mathbf{\Theta }\right)={\mathit{\zeta }}_{\mathrm{local}}+\mathit{\eta }\cdot {\mathit{\zeta }}_{\mathrm{global}},\end{array}$$

where Θ refers to the learnable arguments for each layer of the deep reconstruction model. η denotes the balancing factor to adjust the ζ_local and ζ_global. In this work, we fixed this factor as 0.1 during the training procedure.

After building up this unified loss function, the presented reconstruction model employs the Adam algorithm as the gradient descent strategy. The number of batch sizes in this model is fixed as 128 for network training (Shi et al., 2020). The total epochs and initial learning rate are determined as 300 and 0.001, respectively. Starting every 30 epochs, the learning rate is degraded through a decay coefficient of 0.5 (Zhang et al., 2018b). The training and testing procedure of the proposed model are implemented by the PyTorch platform. The software environment is listed as follows: Python 3.7.4 language, Windows 10 operating system, and PyCharm 2019 integrated development environment (IDE). The final soil moisture products are exported as a hierarchical data format, which contains both the original and reconstructed soil moisture data.

4 Experimental results and validation

In this section, we provide the experimental results and related validation results to testify the availability of the presented framework. Through this framework, we finally generate the seamless global daily AMSR2 soil moisture long-term products from 1 January 2013 to 31 December 2019. The daily soil moisture products are saved in NetCDF4 format. This dataset can be directly downloaded at https://doi.org/10.5281/zenodo.4417458 (Zhang et al., 2021) for free use. Codes are released at https://github.com/qzhang95/SGD-SM (last access: 27 February 2021).

We firstly give two sample seamless reconstruction results of global time-series soil moisture products. The original and reconstructed results are both given for comparisons. Later, to further validate the effectiveness of these products, three verification methods are employed as follows:

1. in situ validation,

2. time-series validation, and

3. simulated missing-region validation.

In situ validation is utilised to compare the reconstructed soil moisture with original AMSR2 soil moisture through the selected in situ sites from the spatial prospect. In situ shallow-depth soil moisture sites can be employed as the ground truth to validate the reconstruction satellite soil moisture products. Time-series validation is employed for evaluating the time-series continuity from the temporal prospect. Soil moisture time-series scatter can obviously reveal the annual periodic variations for time-series validation. Simulated missing-region validation is used to evidence the soil moisture consistency from the spatial prospect. It can verify the spatial consistency between the valid and invalid soil moisture regions.

4.1 Experimental results

As displayed in Figs. 5a–h and 6a–h, original and reconstructed global daily time-series AMSR2 soil moisture products between 1–4 June 2019 and 1–4 October 2016 are given as the sample results, respectively. The left column lists the original incomplete soil moisture results, and the right column lists the corresponding complete soil moisture results after reconstruction by the proposed method from 1 to 4 June 2019 and 1 to 4 October 2016. We ignore the coverage of Antarctica and most of Greenland, because the satellite soil moisture data within these regions are perennially missing.

From the spatial dimension, the reconstructed global soil moisture products are consistent between invalid regions and their adjacent valid regions in Figs. 5 and 6. Especially around the high-value areas and low-value areas, the spatial information is consecutive without obvious reconstruction boundary effects such as in Africa, Australia, and Europe in Figs. 5 and 6.

From the temporal dimension, although the incomplete time-series daily results are highly similar and correlative, there are still some variations and differences between each other. The proposed method performs well when preserving consistent temporal information and predicting specific temporal information in Figs. 5 and 6.

Figure 5 Original and reconstructed global daily SM results between 1 and 4 June 2019.

Figure 6 Original and reconstructed global daily SM results between 1 and 4 October 2016.

4.2 In situ validation

In situ shallow-depth soil moisture sites can be employed as the ground truth to validate the reconstructed satellite soil moisture products. We select 113 soil moisture stations (0–5 cm) through ISMN between 1 January 2013 and 31 December 2019. Nine in situ soil moisture sites and the corresponding reconstruction data within invalid regions are then contrasted as the scatter plots in Fig. 7a–i, respectively. The horizontal axis stands for the in situ soil moisture value. Meanwhile the vertical axis represents the reconstructed soil moisture value. It should be highlighted that due to the lack of recorded data between 2013 and 2019, most in situ values are incomplete with different point numbers. As shown in Fig. 7a–i, the correlation coefficient (R) indexes are distributed between 0.679 and 0.754. The root-mean-square error (RMSE) and mean absolute error (MAE) indexes are distributed from 0.026 to 0.134 and from 0.021 to 0.107, respectively.

Figure 7 Scatters of the in situ reconstructed soil moisture values within selected COsmic-ray Soil Moisture Observing System (COSMOS) stations.

In addition, we compare the reconstructed with original AMSR2 soil moisture products through the selected 113 in situ sites, as listed in Table 1. The averaged R, RMSE, and MAE of the original and reconstructed soil moisture products are 0.685 (0.689), 0.097 (0.093), and 0.079 (0.077), respectively. Overall, the accuracy of reconstructed soil moisture products is generally accorded with the original products. The differences of these indexes R, RMSE, and MAE are minor between the original and reconstructed soil moisture results in Table 1. To some degree, this validation ensures the reliability and availability of the proposed seamless global daily AMSR2 soil moisture products. In addition, the in situ validation results in the Tibetan Plateau region (Qiu et al., 2020; P. Zhang et al., 2020) are listed in the Supplement.

Table 1 Comparisons between original and reconstructed soil moisture products.

4.3 Time-series validation

To further validate the reconstructed soil moisture results, time-series variations in both original and reconstructed results are stacked in six points around the six continents: Africa (0.375^∘ N, 36.875^∘ E), Europe (49.375^∘ N, 35.125^∘ E), Asia (38.125^∘ N, 117.375^∘ E), North America (39.875^∘ N, 106.125^∘ W), South America (15.125^∘ S, 52.625^∘ W), and Australia (30.125^∘ S, 150.375^∘ E). As described in Fig. 8a–f, the horizontal axis stands for the daily time-series date between 1 January 2013 and 31 December 2019. The vertical axis represents the soil moisture value. The blue points refer to the original valid soil moisture daily results, and the red forks stand for the reconstructed invalid soil moisture daily results in Fig. 8.

As depicted in Fig. 8a–f, most of the soil moisture time-series scatters can obviously reveal the annual periodic variations. The reconstructed soil moisture results generally show a fine temporal consistency with the original soil moisture results in different areas. Related low soil moisture values mostly existed in the drought season of winter with frozen lands such as in Fig. 8d. Related high soil moisture values were mainly generated in the moist season of summer with more rainy days, especially in Fig. 8b, d, and f.

Overall, compared with the whole original variation tendency between 2013 and 2019, the generated long-term seamless global daily AMSR2 soil moisture products can steadily reflect the temporal consistency and variation. This is significant for time-series applications and analysis. This daily time-series validation also demonstrates the robustness of the presented method and the availability of the established seamless global daily products.

Figure 8 Original and reconstructed time-series results in selected regions.

4.4 Simulated missing-region validation

In addition to the time-series consistency in Sect. 4.3, the spatial continuity is also important for the reconstructed seamless soil moisture products. Therefore, to further evidence this key point, we carry out the simulated missing-region validation in this subsection. Based on the original soil moisture products, six simulated square missing regions are performed on six continents. In this way, we can easily compare the reconstructed SM regions with original SM regions, to validate the 2D spatial continuity of the proposed SGD-SM products. We select four dates of the long-term soil moisture products: 25 July 2013, 25 July 2015, 25 July 2017, and 25 July 2019 as the simulated objects. For example, original and reconstructed results with simulated missing regions in 25 July 2019 are depicted in Fig. 9a and b, respectively. The simulated missing regions can be clearly observed in Fig. 9a around the six continents. Detailed original and reconstructed spatial information of four simulated patches in 25 July 2015 are displayed in Fig. 10. Table 2 gives the evaluation index (R, RMSE, MAE) of the simulated patches between 2013 and 2019. Then the original reconstructed scatters of simulated regions in 2013, 2015, 2017, and 25 July 2019 are listed in Fig. 11a–b, respectively.

As shown in Fig. 9a and b, the reconstructed invalid regions are consecutive between the original valid regions. And in the simulated missing regions, the spatial texture information is also continuous without obvious boundary reconstruction effects in Fig. 9b. To better distinguish the spatial details of reconstructed soil moisture, we select four enlarged patches in simulated regions in Fig. 10. It can be clearly observed that the reconstructed patches perform with high consistency with the original patches, as displayed in Fig. 10.

In addition, the reconstructed soil moisture patches in simulated missing regions show high reconstruction accuracy, whose R values are distributed between 0.963 and 0.974 in Table 2 and Fig. 11a–d. RMSE and MAE values also perform well with 0.065 to 0.073 m³/m³ and 0.044 to 0.052 m³/m³ in Table 2 and Fig. 11a–d, respectively. Overall, this simulated missing-region validation manifests the reconstruction ability of spatial information continuity.

Table 2 Evaluation indexes of the simulated patches between 2013 to 2019.

Figure 9 Original and reconstructed results with simulated missing regions in 25 July 2019.

Figure 10 Detailed original and reconstructed spatial information of four simulated patches on 25 July 2015.

Figure 11 Original and reconstructed scatter of simulated regions in 2013, 2015, and 2017 and on 25 July 2019.

5 Discussion

5.1 Comparisons with time-series averaging

As mentioned in Sect. 1, some simple strategies such as time-series averaging can also be employed for synthesising the complete soil moisture products. Therefore, we perform the comparisons between the time-series averaging approach and the presented method, to further validate the effectiveness and rationality of our dataset and framework. In terms of the time-series averaging method, it averages the time-series daily soil moisture data to reconstruct gap regions. The original soil moisture result, time-series averaging result, and the proposed reconstruction result on 10 September 2016 are shown in Fig. 12a–c, respectively. Three reconstructed regions are marked with a black circle in Fig. 12b and c. The evaluation index comparisons between the time-series averaging method and proposed method are listed in Table 3 through the corresponding in situ data validations.

Figure 12 Original and time-series averaging and proposed global soil moisture results on 10 September 2016.

Table 3 Evaluation index (R, RMSE, MAE) comparisons between the time-series averaging and proposed method. The better indexes are marked in bold.

As displayed in the black circled regions of Fig. 12b and c, we can clearly distinguish the spatial discontinuity in the time-series averaging result. Reversely, the proposed method performs better on spatial continuity between the valid and invalid regions. The evaluation indexes R, RMSE, and MAE also manifest the superiority of the presented approach, compared with the time-series averaging method in Table 3. The main reason is that daily soil moisture products show temporal differences, and the time-series averaging strategy cannot use the 2D spatial information and ignores these temporal differences. Therefore, it reflects the obvious “boundary difference effect”, especially in the circled regions of Fig. 12b. This also reveals the limitations and shortages of the time-series averaging method. Conversely, the proposed method jointly utilises both spatial and temporal information of these time-series soil moisture products. Further, it can better exploit the deep spatio-temporal feature for soil moisture data reconstruction. Overall, this discussion demonstrates the superiority of the proposed framework for daily time-series product reconstruction, especially compared with the time-series averaging strategy.

5.2 Uncertainty analysis of the SGD-SM products

Uncertainty analysis is important for quantitative remote sensing products. The uncertainties in this generated SGD-SM product can be classified as three types: (1) the errors of the original AMSR2 SM product, (2) the meteorological factors such as precipitation and snowfall, and (3) the generalisation of the proposed reconstruction model. Detailed descriptions of these three uncertainties are listed as follows.

1. The errors of original AMSR2 SM product. The proposed SGD-SM product is generated based on the original AMSR2 SM product. This original AMSR2 SM product also contains errors, due to the satellite sensor imaging and SM retrieval algorithm. As shown in Table 1, the R, RMSE, and MAE evaluation indexes of the original AMSR2 SM product are 0.687, 0.095, and 0.078, respectively. These errors are also inevitably transmitted into the generated SGD-SM product.

2. The meteorological factors. SGD-SM relies on the temporal continuity and spatial consistency for daily SM gap-filling. Nevertheless, if the unusual meteorology occurs in a single day, such as precipitation and snowfall, it may destroy the above assumption and influence the reconstruction effects. This uncertainty can be noticed in time-series validation, especially for the rainy season.

3. The generalisation of the proposed reconstruction model. In this work, we train the proposed network through selecting complete soil moisture patches. In addition, the simulated masks are also chosen from the daily soil moisture products. However, there are still differences between the training data and testing data, such as land cover type, mask size, and so on. This uncertainty may disturb the generalisation of the proposed reconstruction model, to some degree.

6 Data availability

This dataset can be downloaded at https://doi.org/10.5281/zenodo.4417458 (Zhang et al., 2021).

7 Conclusions

In this work, aiming at the spatial incompleteness and temporal discontinuity, we generate a seamless global daily (SGD) AMSR2 soil moisture long-term product from 2013 to 2019. To jointly utilise spatial and temporal information, a novel spatio-temporal partial CNN is proposed for AMSR2 soil moisture product gap-filling. The partial 3D-CNN and global–local loss function are developed for better extracting valid region features and ignoring invalid regions through data and mask information. Three validation strategies are employed to testify the precision of our seamless global daily products as follows: (1) in situ validation, (2) time-series validation, and (3) simulated missing-region validation. Evaluation results demonstrate that the seamless global daily AMSR2 soil moisture dataset shows high accuracy, reliability, and robustness.

Although the proposed framework performs well when generating this seamless global daily soil moisture dataset, some drawbacks and limitations still need to be overcome, especially on multi-source data fusion, spatio-temporal information extraction, and deep learning model optimisation. In our future work, we will introduce multi-source information fusion into the proposed model, such as precipitation and snowfall. The proposed reconstruction model will be increasingly improved by means of more powerful units and structures. In addition, we will consider more soil moisture products in our future work such as AMSR-E, SMOS-IC, SMAP, and so on.