the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 16 Jan 2025
A seamless global daily 5 km soil moisture product from 1982 to 2021 using AVHRR satellite data and an attention-based deep learning model
Abstract. Soil moisture (SM) data records longer than 30 years are critical for climate change research and various applications. However, only a few such long-term global SM datasets exist, and they often suffer from large biases, low spatial resolution, or spatiotemporal incompleteness. Here, we generated a consistent and seamless global SM product from 1982 to 2021 using deep learning (DL) by integrating four decades of Advanced Very High Resolution Radiometer (AVHRR) albedo and land surface temperature products with multi-source datasets. Considering the temporal autocorrelation of SM, we explored two types of DL models that are adept at processing sequential data, including three long short-term memory (LSTM)-based models, i.e., the basic LSTM, Bidirectional LSTM (Bi-LSTM), and Attention-based LSTM (AtLSTM), as well as a Transformer model. We also compared the performance of the DL models with the tree-based eXtreme Gradient Boosting (XGBoost) model, known for its high efficiency and accuracy. Our results show that all four DL models outperformed the benchmark XGBoost model, particularly at high SM levels (> 0.4 m3 m-3). The AtLSTM model achieved the highest accuracy on the test set, with a coefficient of determination (R2) of 0.987 and root mean square error (RMSE) of 0.011 m3 m-3. These results suggest that utilizing temporal information as well as adding an attention module can effectively enhance the estimation accuracy of SM. Subsequent analysis of attention weights revealed that the AtLSTM model could automatically learn the necessary temporal information from adjacent positions in the sequence, which is critical for accurate SM estimation. The best-performing AtLSTM model was then adopted to produce a four-decade seamless global SM dataset at 5 km spatial resolution, denoted as the GLASS-AVHRR SM product. Validation of the GLASS-AVHRR SM product using 45 independent International Soil Moisture Network (ISMN) stations prior to 2000 yielded a median correlation coefficient (R) of 0.73 and unbiased RMSE (ubRMSE) of 0.041 m3 m-3. When validated against SM datasets from three post-2000 field-scale COsmic-ray Soil Moisture Observing System (COSMOS) networks, the median R values ranged from 0.63 to 0.79, and the median ubRMSE values ranged from 0.044 to 0.065 m3 m-3. Further validation across 22 upscaled 9 km Soil Moisture Active Passive (SMAP) core validation sites indicated that it could well capture the temporal variations in measured SM and remained unaffected by the large wet biases present in the input European reanalysis (ERA5-Land) SM product. Moreover, characterized by complete spatial coverage and low biases, this four-decade, 5 km GLASS-AVHRR SM product exhibited high spatial and temporal consistency with the 1 km GLASS-MODIS SM product, and contained much richer spatial details than both the long-term ERA5-Land SM product (0.1°) and European Space Agency Climate Change Initiative combined SM product (0.25°). The annual average GLASS-AVHRR SM dataset from 1982 to 2021 is available at https://doi.org/10.5281/zenodo.14198201 (Zhang et al., 2024), and the complete product can be freely downloaded from https://glass.hku.hk/casual/GLASS_AVHRR_SM/.
Competing interests: Some authors are members of the editorial board of Earth System Science Data
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.- Preprint
(3213 KB) - Metadata XML
- BibTeX
- EndNote
Status: open (until 23 Jun 2025)
-
CC2: 'Comment on essd-2024-553', SHAOBO SUN, 19 Feb 2025
reply
General commentsThe authors developed a consistent and seamless global soil moisture product over 1982-2021 using deep learning models and existing SM and auxiliary data. Validation showed that the SM data well capture temporal variability of global SM. Thus, the newly developed, long-term data show potential in wide applications. While, the manuscript needs to be improved.Specific comments
- The authors did not use SM measuremnts to establish a DL model and develop global SM product. I think the developed GLASS-AVHRR SM product is a reprocessed SM product from ERA-Land SM and previous GLASS MODIS SM products. In the abstract, the authors said "we generated a consistent and seamless global SM product from 1982 to 2021 using deep learning (DL) by integrating four decades of Advanced Very High Resolution Radiometer (AVHRR) albedo and land surface temperature products with multi-source datasets."; did not mention use of ERA-Land SM which is much important for generating the SM data (Fig. A2). I think the authors should clearly stated how they generated the GLASS-AVHRR SM product with existing datasets in Abstract.
- Page 3: "including land cover mapping, data fusion and downscaling, andenvironmental parameter retrieval (Yuan et al., 2020). " Please add citations for each research area.
- Table 1: Many other available long-term SM products were not included, such as the GLEAM SM.
- Section 2 datasets: introductions on the datasets used in this section were verbose. I suggest the author simply simplify this section.
- Section 3 methods: please also simplify introductions on the DL/machine learning models, and pay much attentions on discussing results.
- Figure 9: In northern high latitudes, permafrost, snow and ice distribute widely, SM values in these regions are invalid in non-growing seasons. Thus, the SM maps in Jan. should masked these regions.
- Line 615: "model based on deep learning" - using DL models
- Page 28: “a topic of ongoing debate” - citations are needed.
- Discussion: I suggest the authors pay more attention on discussing their results, including uncertainties in the their developed SM data, and importance of the input variables (Fig. 2A). Specially, Fig. 2A shows that elevation exhibits largest influence on predicting SM. While, the ERA-Land SM had small important value. Why?
Citation: https://doi.org/10.5194/essd-2024-553-CC2 -
RC1: 'Comment on essd-2024-553', Anonymous Referee #1, 27 Feb 2025
reply
This study creates a consistent long-term and high-resolution global soil moisture product with comparable accuracy to previous deep learning approaches and demonstrates a clear superiority of the AtLSTM approach over other compared approaches. The findings and datasets are valuable to the community. The reporting is clear. Overall I recommend it for publication after minor revisions. Please see below for comments:
1. The depth of the developed dataset should be noted in the abstract.
2. The training target is a previously developed 1km dataset, and the newly developed dataset mainly provides the advantage of being longer-term and better winter coverage. This would be clearer if the temporal coverage of the Zhang et al. (2023) dataset is stated in p4 lines 117-118.
3. p5 line 146: "8-day temporal resolution interpolated to daily" Please specify which interpolation method was applied.
4. It would be nice to add 1-2 sentences about uncertainty in the sampling strategy of the training target (p9 lines 242-p10 ln 244).
5. p13 Table 4: Should the number of layers be "2" for Bi-LSTM and 1 elsewhere?
6. The sequence lengths in Table 4 span whole year. The testing on p17 only spans 0-29 days and demonstrates stabilized performances at much shorter lengths than 29 days. Why the drastic increase in sequence lengths in production runs? Also, since the input data are 3 discrete years, please specify what values are used to pad the 60 days before the start and after the end of the whole year.
7. p14 lines 366-367: A bit confusing because this reduced-sample testing was not described in the Methods. Could you add a description?
8. p15 Table 5: Is there a specific reason for choosing 0.4 m3/m3 as a threshold for large SM values?
9. Fig. 6: COSMOS dataset is not directly comparable to the developed dataset due to significant depth differences. It is okay as an auxiliary comparison, given that other comparable independent validation datasets are used (1982-1999 ISMN and SMAP validation sites). However, for informative purpose, it would be good to provide boxplots of the depths of the COSMO observations used in the last three columns of Fig. 6 in supplementary info.
10. More on Fig. 6. The currently used 1982-1999 ISMN set provides spatiotemporally independent comparison. The 1982-1999 data at the 715 representative stations would still be temporally independent comparison. Could you include the performance metrics on this subset of data?
Citation: https://doi.org/10.5194/essd-2024-553-RC1 -
CC3: 'Comment on essd-2024-553', SHAOBO SUN, 06 May 2025
reply
Publisher’s note: this comment is a copy of CC2 and its content was therefore removed on 7 May 2025.
Citation: https://doi.org/10.5194/essd-2024-553-CC3 -
RC2: 'Comment on essd-2024-553', Anonymous Referee #2, 06 May 2025
reply
This study used several deep learning models to estimate SM with some optical remote sensing indices, soil properties and DEM. Even though it’s a big work, it lacks sufficient innovation and employs inappropriate evaluation method. Despite the overall scale of the work, the methodological design raises serious concerns.
Major Comments:
Line 135, Table 2: I question the innovation and scientific contribution of using ERA5-Land SM as an input variable in a model that aims to estimate soil moisture. This approach may introduce circular reasoning and undermines the novelty of the proposed method.
Line 380, Table 5: It is inappropriate to use GLASS-MODIS SM as a reference to evaluate the model's estimated SM, as they are not independent. Since variables such as LST and albedo were also used in the model estimation, the evaluation becomes biased, leading to inflated performance metrics (e.g., R² > 0.98). This dependence compromises the reliability of the validation.
Line 20: The five deep learning models yielded very similar results, with R² values ranging from 0.982 to 0.987. Therefore, the claim that these models “…effectively enhance…” SM estimation is not well supported.
Minor Comments:
Line 60: There are additional remote sensing-based SM products that should be referenced, including but not limited to: Cheng et al. (2023); Guevara, Taufer, & Vargas (2021); and Zheng, Jia, & Zhao (2023).
Line 190: The time period referred to as "period1" is 2000–2018. Why were the years 2019–2021 excluded from the analysis?
Line 355: The statement “two widely used long-term global SM products” should specify which products are being referred to for clarity.Cheng, F., Zhang, Z., Zhuang, H., Han, J., Luo, Y., Cao, J., . . . Tao, F. (2023). ChinaCropSM1 km: a fine 1 km daily soil moisture dataset for dryland wheat and maize across China during 1993–2018. Earth System Science Data, 15(1), 395-409. doi:10.5194/essd-15-395-2023
Guevara, M., Taufer, M., & Vargas, R. (2021). Gap-free global annual soil moisture: 15 km grids for 1991–2018. Earth System Science Data, 13(4), 1711-1735. doi:10.5194/essd-13-1711-2021
Zheng, C., Jia, L., & Zhao, T. (2023). A 21-year dataset (2000-2020) of gap-free global daily surface soil moisture at 1-km grid resolution. Sci Data, 10(1), 139. doi:10.1038/s41597-023-01991-wCitation: https://doi.org/10.5194/essd-2024-553-RC2 -
RC3: 'Comment on essd-2024-553', Anonymous Referee #3, 08 May 2025
reply
The authors compared multiple deep learning methods and selected the most suitable approach to generate a 5 km-resolution soil moisture product based on AVHRR data, spanning four decades. Overall, this is a well-written manuscript, and the scope of the study aligns well with the aims and scope of Earth System Science Data. However, the current version of the manuscript appears overly technical in its presentation. The authors should address the following major and specific comments before the manuscript can be considered for publication.
General comments
- While the description of the deep learning methods is sufficiently detailed, the treatment of the predictor variables requires substantial clarification and improvement. It remains unclear how these variables were selected, particularly the rationale for using ERA5-Land data as model inputs. In my view, the relatively strong performance of GLASS-AVHRR in capturing seasonal variations, especially for the R metric, may be partially attributed to this. The authors should consider producing an independently driven soil moisture product, rather than relying on predictors that may introduce redundancy or circular reasoning.
- I remain concerned about the generalization capability of the developed deep learning model, particularly considering the strong spatial autocorrelation inherent in the current training, validation, and test data split. Furthermore, the results presented in Table 5 indicate that model performance deteriorates when soil moisture exceeds 0.4 m³/m³, which the authors attribute to a lack of training samples. This observation indirectly suggests that the model performs poorly in unseen regions or conditions, highlighting its limited generalization ability. The authors should explore alternative data splitting strategies to rigorously assess the robustness and generalizability of their approach.
- Figure 2 in the Methods section is unnecessarily technical and lacks clear justification. A concise textual explanation that hyperparameter tuning was performed would be sufficient. Moreover, both Figures 1 and 2 fail to clearly illustrate how the spatiotemporal training was implemented, especially considering that LSTM and related models are specifically designed to capture temporal dependencies. The authors should provide a more explicit and structured explanation of how time-series characteristics were incorporated into the model training, without ignoring the spatial autocorrelation issue raised above.
- In Figure 9, I noticed that the GLASS-AVHRR estimates provide coverage during the winter season, unlike the CCI product. The authors should clarify what measures were taken to ensure the reliability of these estimates under winter conditions, particularly in areas covered by snow, ice, or water. An additional seasonal evaluation focusing specifically on winter performance would be welcome and could strengthen the credibility of the product. In addition, all products in Figure 9 should be masked over permanent water bodies, where soil moisture retrieval is not valid. This issue is especially noticeable in Quebec, Canada.
- The authors should provide more reasonable explanations for some of the evaluation results. For example, why does GLASS-AVHRR perform worse than ERA5-Land in terms of the R metric over COSMOS-Europe? In addition, I do not fully agree with the explanation provided for the strong dry bias observed in COSMOS-UK. ERA5-Land shows a consistently wet bias across different COSMOS networks, which raises the question of whether the developed product itself suffers from instability in different regions.
Specific comments:
L61: This may also be related to the uneven spatial distribution of the input meteorological data, particularly the limited observational coverage in tropical regions.
L64: also 10.1016/j.rse.2022.112921; 10.1016/j.rse.2022.113272
2 Datasets: Predictor variables lack of vegetation-related variables
Table 2: As noted above, it is inappropriate to use ERA5-Land-derived soil moisture as an input when the objective is to analyze or generate a new soil moisture product.
L146: What kind of method?
L156: see 10.1016/j.rse.2023.113721
L175: This also means that validation with ISMN is not independent.
L196: The meaning of the label "G" should be clearly defined.
L201: How do the authors justify the appropriateness of using COSMOS observations, which primarily sense deeper soil moisture, for evaluating surface soil moisture products?
Table 3: Are these COSMOS networks intercalibrated?
Sections 3.3 and 3.4 can be simplified
L377: The term "underestimate" should be used with caution, as all reference datasets represent proxy observations rather than true ground-truth measurements at the grid scale.
Figure 3: Lack of units
L506-507: Why doesn't ERA5-Land have this problem?
Figure 6: These three datasets, including ERA5-Land, GLASS-MODIS and GLDAS-AVHRR, have different spatial resolutions. Please clarify how the spatial mismatch was addressed when using different in-situ observations.
L529: Why doesn't ERA5-Land have this problem?
Figure 10: All products should be masked out over lake areas where soil moisture retrievals are not valid.
Citation: https://doi.org/10.5194/essd-2024-553-RC3 -
CC4: 'Comment on essd-2024-553', Noemi Vergopolan, 13 May 2025
reply
Publisher’s note: this comment is a copy of RC4 and its content was therefore removed on 15 May 2025.
Citation: https://doi.org/10.5194/essd-2024-553-CC4 -
RC4: 'Comment on essd-2024-553', Noemi Vergopolan, 14 May 2025
reply
This paper introduces a dataset and approaches integrating AVHRR albedo and land surface temperature using an LSTM-based deep learning model to reconstruct soil moisture from 1982 to the present from GLASS-MODIS. The paper is well written, and the methodology is clear and well-detailed, but the paper does not highlight the novelty of the approach and how it differentiates from other efforts in the field. One major concern is that GLASS-MODIS already uses ERA5-Land, MERIT-DEM, and Soil-Grids as inputs; as such the proposed DL models are simply acting as surrogate models to GLASS-MODIS. The authors should demonstrate that AVHRR and the proposed DL model substantially add value to the predictions of soil moisture. As it reads, the added value of the approach is limited to extending the soil moisture time series to the AVRHH time series, which is helpful to evaluate long-term trends, but I wonder how it adds value beyond a CDF matching bias correction, for example.
Major:
GLASS-MODIS already uses ERA5-Land, MERIT-DEM, and Soil-Grids, which are also input data for your approach in this paper. How do you ensure your DL model is actually learning anything from GLASS-AVHRR and not just a surrogate for GLASS-MODIS learning directly the relationship from ERA5-Land, MERIT-DEM and SoilGrids. In other words, the if the target data is not independent from the input data, your DL model is just a surrogate for the existing GLASS-MODIS model. Two ways to quantitatively test for that is to 1) check how statistically significant is the difference in the accuracy of your trained DL models with respect to GLASS-MODIS when validating the approach with in-situ observations, and 2) by testing how different is your DL accuracy if trained only on ERA5-Land, MERIT-DEM and Soil-Grids for the past period (1982-1999). These experiments would help elucidate how the proposed model is actually learning new things from the AVHRR data or just mimicking GLASS-MODIS from ERA5-Land, MERIT-DEM, and Soil-Grids.
Moderate:
R2 0.987 indicates that probably the DL is overfitting or there is contamination between the training and testing datasets used. In fact, when using the model to produce the GLASS-AVHRR dataset accuracy drops from R 0.63 to 0.79
Minor:
I appreciate how the authors ensured that testing grids are within 25km distance from training grids. This prevents the testing sample is contaminated with inputs from the training datasets (since the resolution of precipitation input data from ERA5 into ERA5-Land runs is about 25km resolution).
Please add GLASS-MODIS to Figure 6 for the COSMOS comparisons. Please include marks indicating whether the difference between the accuracy of the SM data products is statistically significant.
Please add GLASS-MODIS to Table 6, which will be best displayed in landscape format. Also, please mark in bold the best statistically significant data product/metric for each site.
Please add GLASS-MODIS to Figure 7.
Citation: https://doi.org/10.5194/essd-2024-553-RC4
Data sets
A seamless global 5 km surface soil moisture product from 1982 to 2021 Yufang Zhang, Shunlin Liang, Han Ma, Tao He, Feng Tian, Guodong Zhang, and Jianglei Xu https://doi.org/10.5281/zenodo.14198201
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 606 | 99 | 17 | 722 | 10 | 8 |
- HTML: 606
- PDF: 99
- XML: 17
- Total: 722
- BibTeX: 10
- EndNote: 8
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1