The GLASS product suite has been employed in various applications owing to its long-term coverage, spatial continuity, high spatial resolution, and accuracy (Liang et al., 2021). Here, the latest version of GLASS albedo, LST, and LAI products served as the primary inputs to the ensemble learning model. Specifically, the GLASS V6 LAI product (500 m resolution) was generated from six MODIS 8 d surface reflectance bands of MOD09A1 using a bidirectional long short-term memory deep learning model (http://www.glass.umd.edu, last access: 12 May 2023) (Ma and Liang, 2022). Notably, this product is relatively more accurate than the 250 m GLASS LAI estimated from two bands of MOD09Q1. The all-sky 1 km GLASS LST was produced by integrating multiple datasets from MODIS, reanalysis, and in situ LST measurements using a random forest model (Li et al., 2021). Daily global LSTs averaged from instantaneous GLASS LST products were used here, which will be also released at http://www.glass.umd.edu. The gap-free GLASS albedo products were generated using a combination of a direct-estimation algorithm (Qu et al., 2014) and a spatiotemporal filtering scheme (Liu et al., 2013). Namely, the black-sky visible, near-infrared, and shortwave albedo data extracted from the GLASS V42 albedo products were used in the present study (http://www.glass.umd.edu).

ERA5 provides a range of global atmospheric, terrestrial, and oceanic variables from 1950 to present at 31 km spatial resolution (Hersbach et al., 2020). Specifically, ERA5-Land is an enhanced global land reanalysis dataset obtained by downscaling the atmospheric forcing derived from the reanalysis of ERA5 to a native resolution of approximately 9 km (Muñoz-Sabater et al., 2021). ERA5-Land includes hourly estimates of volumetric soil moisture at four soil layers and a grid spacing of 0.1∘ (https://cds.climate.copernicus.eu/, last access: 12 May 2023). In the present study, the top layer (0–7 cm) ERA5-Land soil moisture were used to match the shallow observation depths of optical satellites. The daily average soil moisture was calculated and resampled to 1 km before being used as an input variable of the ensemble learning model.

Topography and soil properties, which can be treated as static variables due to their relatively slow rate of change over the short term, have an important influence on the spatial variations of soil moisture at finer scales. The global terrain dataset used in the study here was the high-accuracy MERIT DEM with a spatial resolution of 3 arcsec (∼ 90 m at the Equator). The MERIT DEM integrates several spaceborne DEMs after eliminating their inherent primary error components, including speckle noise, stripe noise, absolute bias, and tree height bias (http://hydro.iis.u-tokyo.ac.jp/~yamadai/MERIT_DEM/, last access: 12 May 2023) (Yamazaki et al., 2017). After deriving the elevation, aspect, and slope from the MERIT DEM, these topographic variables were resampled to 1 km and used as input features for the model. Alternatively, soil texture was derived from the SoilGrids V2.0 product at 250 m resolution (https://www.isric.org/explore/soilgrids, last access: 12 May 2023). SoilGrids uses > 240 000 soil profile measurements and > 400 environmental covariates worldwide to train machine learning models and produce global soil property maps across six depth intervals (Poggio et al., 2021). Recent studies have shown that the SoilGrids product has both higher resolution and enhanced accuracy compared to other soil datasets at the global scale (Dai et al., 2019), in addition to the ability of soil texture data to improve the bias and root mean square error (RMSE) of downscaled soil moisture products (Das et al., 2019). Accordingly, the mean contents of sand, silt, and clay were extracted for the first soil layer (0–5 cm) from the SoilGrids database and resampled to 1 km.

The ISMN aims to establish and maintain a global database of in situ soil moisture measurements for the validation and improvement of satellite-based and modeled soil moisture products. Currently, it consists of 73 networks with over 2800 soil moisture stations worldwide, providing quality-controlled and harmonized datasets collected from monitoring networks and field experiments (Dorigo et al., 2021). Here, data for the period from 2000–2018 were obtained (https://ismn.earth/, last access: 12 May 2023), and only stations with a sensing depth of < 5 cm were selected to match the observation depth of remotely sensed datasets. Soil moisture records were then screened according to the quality flags provided with the ISMN dataset (Dorigo et al., 2013), before being used as the training target for the machine learning model. The spatial distribution of the representative ISMN soil moisture stations selected using the TC method described in Sect. 3.2 is displayed in Fig. 1. The number and percentage of representative stations for each land cover type and climate class, which are calculated by using the 500 m MODIS land cover type product (Friedl and Sulla-Menashe, 2019) and the 1 km Köppen–Geiger climate classification dataset (Cui et al., 2021), respectively, are also shown in Table 3.

Four soil moisture monitoring networks that were not included in the ISMN database were used to assess the model's ability to capture temporal variations in soil moisture over unknown area (Fig. 1). The YA and YB subnetworks are both part of the Yanco soil moisture network, located in a semiarid agricultural region of the Murrumbidgee River basin, Australia, with a flat topography and elevation spanning 117–150 m (Yee et al., 2017). There are 13 and 11 stations in the YA and YB subnetworks, respectively, distributed across two 9 × 9 km areas, and soil moisture observations from these stations can be downloaded from the Oznet Hydrological Monitoring website (http://www.oznet.org.au, last access: 12 May 2023) (Smith et al., 2012). Two other micronets (Fort Cobb and Little Washita) are located in southwestern Oklahoma, USA, and are characterized by a humid subtropical climate (Starks et al., 2014). The primary land cover types are cropland and rangeland, and the topography is moderately rolling (Bindlish et al., 2009). Currently, there are 15 and 20 operational stations in the Fort Cobb and Little Washita networks, respectively, for which soil moisture datasets can be accessed through the Grazinglands Research Laboratory (https://ars.mesonet.org/, last access: 12 May 2023). These four dense soil moisture networks have been used extensively to either validate or calibrate satellite soil moisture products (Ma et al., 2021; Colliander et al., 2017; Chan et al., 2018).

To further validate the spatiotemporal performance of the derived 1 km soil moisture product here, two additional microwave-based products were selected for comparison. The first product is the high-resolution SMAP/Sentinel-1 SPL2SMAP_S dataset, which contains the first global 1 km soil moisture product that was publicly released in the past (Table 1). It has a temporal resolution of 6–12 d and can be downloaded from the National Snow and Ice Data Center at 1 and 3 km resolutions (https://nsidc.org/data/spl2smap_s, last access: 12 May 2023). According to Das et al. (2019), the average unbiased RMSE (ubRMSE) values achieved by both the 1 and 3 km SPL2SMAP_S products over sparse soil moisture networks were approximately 0.05 m3 m-3. Considering that the SPL2SMAP_S baseline algorithm generally shows higher validation accuracy than the optional algorithm (directly disaggregating the SMAP 9 km soil moisture product) and that the AM (descending orbit combination) soil moisture retrievals are more accurate than their APM (a.m. or p.m.) equivalents (descending or ascending orbits combination) (Xu, 2020), the baseline AM soil moisture field “disagg_soil_moisture_1km” data were extracted from the SPL2SMAP_S 1 km data group and used for comparison. The second product is the CCI global soil moisture dataset released by the ESA, with a grid spacing of 0.25∘ and daily temporal resolution, which combines various passive and active microwave soil moisture products into a harmonized record with improved spatiotemporal coverages and has been fully validated across numerous global applications (Gruber et al., 2019; Dorigo et al., 2017). Specifically, the combined (active and passive) soil moisture product from CCI V6.1 was used here (https://esa-soilmoisture-cci.org/data, last access: 12 May 2023).

Soil moisture is characterized by high spatiotemporal variability, and its distribution is influenced by a range of environmental factors across different scales, such as climate, geographical conditions, soil properties, and surface coverage (Crow et al., 2012; Luo et al., 2022). Here, high-accuracy, spatiotemporally continuous GLASS products, including LST, albedo, and LAI, were used to provide surface temperature, spectral information on soil and vegetation, and information related to vegetation type and density. Considering the impact of topography and soil properties on soil moisture, topographic and soil texture fraction variables were extracted from the MERIT DEM and SoilGrids products, respectively. Additionally, the 0.1∘ ERA5-Land reanalysis soil moisture product was used to provide background soil moisture information. By utilizing an ensemble machine learning model, various variables extracted from these multisource datasets were integrated so that different environmental factors affecting soil moisture could be accounted for, and then soil moisture at fine scales could be estimated.

Figure 2 shows a flowchart of the proposed 1 km, spatiotemporally continuous soil moisture estimation framework. Prior to the training phase, the TC method and the other two long-term soil moisture datasets (ERA5-Land reanalysis and ESA CCI soil moisture products) were adopted for selecting the representative soil moisture stations, considering the scale difference between point-scale soil moisture measurements collected by ISMN stations and GLASS products (the detailed selection procedure is presented in Sect. 3.2). Then, multiple variables were extracted from the corresponding input datasets and spatiotemporally collocated with the in situ soil moisture measurements from the representative stations between 2000 and 2018. Specifically. the black-sky visible, near-infrared, shortwave albedo, LAI, and LST were extracted from the three GLASS products, based on the geographic locations of stations. Each of these variables, together with topographic and soil texture fraction variables, and the coarse-scale reanalysis soil moisture were put into the XGBoost model, which was chosen to simulate the nonlinear relationship between multiple input features and in situ soil moisture (the target variable). Lastly, those multisource input datasets were resampled to 1 km and then put into the developed XGBoost model for predicting the global 1 km spatiotemporally continuous soil moisture product (GLASS SM). Moreover, the GLASS SM product was evaluated against four independent soil moisture datasets and then compared the SPL2SMAP_S and CCI soil moisture products for spatiotemporal consistency analyses.

As mentioned above, in situ soil moisture data from the ISMN stations were employed as the target variable to train the XGBoost model, which was then used to predict soil moisture product at 1 km resolution. The underlying assumption was that the measured soil moisture at these point-scale stations is representative of the average moisture status of the corresponding 1 km pixel; however, because of the high spatiotemporal variability of soil moisture, this assumption is not always upheld. Accordingly, the TC method, which has been widely applied to analyze the coarse-scale spatial representativeness of in situ soil moisture dataset (Gruber et al., 2013; Molero et al., 2018), was adopted here to select the most representative stations. Specifically, TC is an error analysis method proposed by Stoffelen (1998) employing three collocated datasets to address large uncertainties in wind speed measurements. TC has also been widely used in the evaluation of satellite soil moisture products given the limited number of core validation sites at the satellite footprint scale (Zheng et al., 2022). The commonly used error model for TC analysis is defined in Eq. (1): 1Xi=αi+βiθ+εi, where Xi refers to the three collocated soil moisture observations; θ refers to the unknown true value of soil moisture; αi and βi are the additive and multiplicative biases of Xi relative to the true value, respectively; and εi is the random additive noise with zero mean. The assumptions underlying this error model and detailed derivation process for the error variance of each dataset can be found in Gruber et al. (2016). Notably, the assumptions made for TC analysis are similar to those made for the correlation coefficient (R) and RMSE (Gruber et al., 2016). To fulfill the independent error requirement of the TC analysis across the three datasets, the ISMN in situ soil moisture, model-based ERA5-Land soil moisture, and CCI combined microwave soil moisture were selected to construct the triplet. Among them, the CCI soil moisture product was selected here rather than other microwave soil moisture products, as it maintains a sufficiently long timescale to cover that of the training samples. The error variance of the ISMN soil moisture dataset, σε2, was then calculated according to Eq. (2): 2σε2=σismn2-Cov(Xismn,Xera)Cov(Xismn,Xcci)Cov(Xera,Xcci), where σismn2 is the variance of the ISMN in situ soil moisture; Cov is the covariance operator; and Xismn, Xera, and Xcci denote the collocated ISMN, ERA5-Land, and CCI soil moisture observations, respectively. Based on TC analysis, McColl et al. (2014) proposed a method called extended triple collocation (ETC) to estimate the correlation coefficient between each dataset and the unknown target variable. Specifically, the ETC correlation coefficient of the ISMN soil moisture dataset, RETC, can be calculated via Eq. (3): 3RETC=sign(±)Cov(Xismn,Xera)Cov(Xismn,Xcci)σismn2Cov(Xera,Xcci), where the sign of RETC was corrected to positive. It is a scaled, unbiased signal-to-noise-ratio metric complementary to σε2. Using the above TC-based metrics and referring to previous studies (Yuan et al., 2020; Anderson et al., 2012), several strict conditions were established to select the most representative ISMN stations: (1) > 500 triplets were available at the station during the period 2000–2018, (2) the R between any two soil moisture datasets in the triplets was > 0.2, (3) the square root of the σε2 calculated for the ISMN soil moisture dataset was < 0.06 m3 m-3, and (4) the RETC between the ISMN soil moisture and the unknown soil moisture true values was > 0.7. A total of 715 representative ISMN soil moisture stations were finally selected, as shown in Fig. 1.

Ensemble machine learning models can be roughly classified into two categories based on how the individual learners are generated: bagging and boosting (Zhou, 2021). For bagging models, the individual learners are constructed independently; whereas for boosting models, learners are constructed iteratively, increasing the weights for the incorrectly classified samples during each round of training. As a representative bagging algorithm, random forest has gained considerable attention in the fields of remote sensing classification and regression over recent decades (Belgiu and Drãguþ, 2016); however, it may suffer from a large prediction bias, especially when the observations are too large or small (Song, 2015). In contrast, boosting models have been shown to reduce both variance and bias and are robust to multicollinearity among predictors (Gislason et al., 2006; Karthikeyan and Mishra, 2021). Accordingly, the present study employed the XGBoost model implemented by Chen and Guestrin (2016) based on a gradient boosting framework (Friedman, 2001). The XGBoost model is advantageous for its scalability, efficiency, and decreased vulnerability to overfitting. Here, the open-source xgboost and Scikit-learn Python packages were used together for model training and hyperparameters tuning, with the grid search method being adopted to determine the optimal parameters. Here, the key hyperparameters of the XGBoost models were finally set to n_estimators (the number of the boosting rounds) = 1000, learning_rate = 0.1, and max_depth (maximum tree depth) = 8.

While most previous soil moisture estimation studies based on machine learning have only used the random validation approach, this study used the three complementary validation strategies to fully evaluate the model performance: random, site independent, and year independent. For the random validation, samples from all soil moisture stations during 2000–2018 were randomly divided into five folds, among which three folds were used for training, one as the validation dataset to tune the hyperparameters of the model, and one as the test dataset to evaluate the model performance. Thus, the samples in the random test dataset may have been from the same station or year as the training or validation datasets. For site-independent validation, all soil moisture stations were again randomly divided into five folds, and samples from one fold were used as the test dataset to evaluate the accuracy of models trained with samples from the other folds, which were used for training and validation. Thus, the location of the samples in the site-independent test dataset is unknown to the model. Similarly, for the year-independent validation, samples from all stations between 2015 and 2018 were selected as the test dataset to evaluate the accuracy of the model trained using samples between 2000 and 2014, to ensure that the observation year was unknown to the model.

In addition to model evaluation, the accuracy of the GLASS SM product generated by the developed model was evaluated. This 1 km soil moisture product was first validated against four independent dense soil moisture networks and then compared with the 1 km SPL2SMAP_S and 0.25∘ CCI soil moisture products for spatiotemporal consistency analyses. Four widely used performance metrics in soil-moisture-related research – the R, bias, RMSE, and ubRMSE (Entekhabi et al., 2010) – are used to evaluate both the models and products against in situ dataset, which can be calculated according to Eqs. (4)–(7): 4R=E[(θest-Eθest)(θtrue-Eθtrue)]σestσtrue,5bias=Eθest-Eθtrue,6RMSE=E[(θest-θtrue)2],7ubRMSE=E{[θest-Eθest-(θtrue-Eθtrue)]2}, where E[.] denotes the mean operator; θtrue and θest represent the in situ soil moisture and corresponding estimated soil moisture; and σtrue and σest refer to the standard deviation of the in situ and estimated soil moisture values, respectively. Note that, while comparing two soil moisture products with similar spatial resolution in Sect. 4.4, the term “root mean square difference (RMSD)” is used, despite the fact that it is also calculated using Eq. (6). Besides, when the large-scale soil moisture product is validated against the point-scale in situ soil moisture dataset, bias often exists between the two datasets because of scale differences, and then R and ubRMSE are typically more informative than RMSE.

Figure 3 shows the overall performance of the XGBoost models developed using all input variables on the random test samples. To analyze the effect of screening soil moisture stations, the accuracies of models developed using all ISMN stations, the representative stations selected using the TC method, and the stations excluded using the TC method (not included in the representative stations) were compared via scatterplots. In general, the random validation accuracy of all three XGBoost models was high, with the bias between the model-predicted and target soil moisture values being close to zero. The accuracy of the models developed using all ISMN stations or the TC-excluded stations was similar for the test samples, with R values of 0.917 and 0.918 and RMSE values of 0.047 and 0.049 m3 m-3, respectively. In contrast, the accuracy of the model developed with the representative stations selected using the TC method was significantly improved for the test samples, with R and RMSE values of 0.941 and 0.038 m3 m-3, respectively. Compared with the other two models, the soil moisture estimates of the XGBoost model developed using representative stations were more concentrated along the 1:1 line. Notably, most of the soil moisture measurements that were nearly saturated (> 0.5 m3 m-3) were excluded after the station screening process (Fig. 3), likely because those high soil moisture samples at point scales were typically under-representative of the mean soil moisture conditions at satellite footprint scales. Meanwhile, the validation accuracy of the ERA5-Land surface soil moisture product was also calculated for all soil moisture samples, as well as those selected by the TC method for comparison. After station screening, the overall R between ERA5-Land reanalysis and in situ soil moisture increased from 0.56 to 0.64, while the RMSE decreased slightly from 0.138 to 0.129 m3 m-3, and the bias remained unchanged at 0.08 m3 m-3. The above performance metrics indicated that representative stations can be effectively selected by using the TC method, and training the XGBoost model with representative stations can significantly improve its validation accuracy on the random test samples.

As can be seen from Table 4, regardless of the type of soil moisture station used during training, model performance on the year-independent test samples (2015 to 2018) decreased significantly compared to that on the random test samples. Among them, the R values of the models trained using all stations and TC-excluded stations were 0.8 and 0.734 for the year-independent test samples, respectively, while the corresponding RMSE increased to 0.07 and 0.084 m3 m-3, respectively. In contrast, the XGBoost model trained using representative stations selected by the TC method achieved the highest accuracy on the year-independent test samples, with R and RMSE values of 0.873 and 0.054 m3 m-3, respectively. Likewise, the performance of the models trained using three different types of stations on the site-independent test samples (randomly selected one-fifth of the total stations) further decreased compared to that of the year-independent test samples. The RMSE values of the models trained using all and excluded stations further increased to 0.093 and 0.106 m3 m-3, respectively, for the site-independent test samples. Alternatively, the XGBoost model trained using representative stations achieved the highest accuracy for the site-independent test samples, with R and RMSE values of 0.715 and 0.079 m3 m-3, respectively. These results suggest that the good performance of the models on the random or year-independent test samples is clearly a result of model overfitting, and their accuracies may degrade significantly when the stations or observation years of the test samples are unknown to them. The relatively lower accuracy achieved by the model on site-independent test samples is least likely to be overfitted and can be regarded as the model's true accuracy. Besides, it appears that increasing the number of stations in the training dataset to account for spatial heterogeneity is more important for improving the models' performance than extending the duration of the measurements to account for temporal dynamics, as also found in a previous study (Zappa et al., 2019). Moreover, training the model with representative stations selected by the TC method can also considerably improve its performance on site- or year-independent test samples, i.e., model performance over unknown time and space.

Figure 4 shows the permutation feature importance results of the XGBoost models trained using representative soil moisture stations, which were calculated separately for the three different types of test samples. The permutation importance of an input feature is commonly measured by the degradation of model accuracy when the feature is randomly shuffled (Breiman, 2001), it can be calculated multiple times across a test dataset and is less likely to be biased towards high-cardinality features. Notably, permutation importance does not reflect a feature's intrinsic predictive value but rather its relative importance to a particular model. For all three types of test samples, ERA5-Land surface soil moisture (SM_era) achieved the highest importance score, indicating that this coarse-scale reanalysis soil moisture product can indeed provide reliable soil moisture background information for the 1 km soil moisture estimation model. Specifically, for both the random and year-independent test samples (Fig. 4a, b), the importance of elevation and soil texture variables (sand, silt, and clay) ranked relatively high, showing that soil properties and topographic factors are important for accurate model predictions when the sample locations are known. In addition, the three GLASS black-sky albedo bands (ABD_vis, ABD_nir, and ABD_short) also achieved relatively high importance scores for both types of samples, likely because surface albedo can reflect the surface energy flux and land cover conditions, which are further correlated to the spatial variation in soil moisture (Long et al., 2019). Meanwhile, the importance scores of GLASS LAI and LST were relatively low for the two sample types, which may be partly attributed to their correlation with some high-ranking variables (e.g., ABD_vis, SM_era). For example, after removing ERA5-Land soil moisture from the models, the importance scores of both GLASS LST and LAI increased significantly. In contrast, for the site-independent test samples (Fig. 4c), the importance of ERA5-Land surface soil moisture (SM_era) further increased relative to other variables. In addition, the importance ranking of GLASS albedo and LST increased remarkably, whereas that of terrain and soil texture-related variables dropped dramatically, suggesting that when the location of the test samples is unknown to the model, variables such as coarse-scale soil moisture, albedo, and LST appear to be more important for accurately predicting soil moisture. Note that the final model was developed using all the representative ISMN stations, and its feature importance results over unknown regions could refer to those calculated on the site-independent test samples.

To further investigate the importance of different types of input variables for the 1 km soil moisture estimation model over unknown space, the validation accuracy of the XGBoost models developed using different combinations of input datasets on the site-independent test samples was also compared. The XGBoost model trained with all input datasets achieved the highest accuracy (Table 5), with R and RMSE values of 0.715 and 0.079 m3 m-3, respectively. After the ERA5-Land soil moisture product was excluded, the model accuracy for the test dataset decreased significantly, with the RMSE value increasing to 0.086 m3 m-3, further reflecting the relatively high importance of the coarse-scale soil moisture background information for the 1 km estimation model derived here. Similarly, after excluding GLASS albedo, LAI, and LST from the input variables, the model trained with the remaining variables showed a marked decrease in accuracy for the test dataset, with R and RMSE values of 0.694 and 0.083 m3 m-3, respectively. This indicates that the information on soil and vegetation reflective properties, surface temperature, and vegetation types and densities provided by GLASS products are also important for the 1 km soil moisture estimation model. Further, the exclusion of terrain or soil texture datasets showed a similar effect on model accuracy, with RMSE values decreasing to 0.082 and 0.083 m3 m-3, respectively, again suggesting the pertinent contribution of these variables to improving the performance of the soil moisture estimation model. Besides, as shown in Table 2, the spatial resolution of most input datasets was within 1 km, except for the ERA5-Land product which had a relatively low spatial resolution (0.1∘). Therefore, the integration of multisource input datasets using a machine learning model can improve not only the model accuracy but also the spatial details of the soil moisture product. Because the XGBoost model trained with all input datasets performed best on the test dataset, all datasets were included in model training during the subsequent experiments.

To explore the causes of decreased 1 km soil moisture estimation model accuracies over unknown time and space, performance metrics of the models were calculated for each station, which were trained using all ISMN or representative soil moisture stations selected by the TC method. To obtain the validation accuracy for each station, a 5-fold cross-validation method was adopted, where the stations were randomly divided into five folds, with samples from four folds used to develop the model, and the accuracy metrics were derived for the remaining fold. This process was repeated five times, until the accuracies of all stations were evaluated. The distribution of performance metrics for the XGBoost model developed using all stations was dispersed across stations, with R values ranging from -1 to 1, and RMSE values ranging from 0.005 to 0.397 m3 m-3 (Fig. 5, Table 6). Although the median of the bias between model predicted and measured soil moisture was 0, the model exhibited a large prediction bias for most stations (from -0.39 to 0.34 m3 m-3), partly contributing to the large RMSE observed at these stations. After removing the prediction bias for each station, the median ubRMSE of the model decreased to 0.055 m3 m-3, compared to the median RMSE of 0.075 m3 m-3. As a comparison, the performance metrics of the ERA5-Land soil moisture product at each ISMN station were also calculated and are displayed in Fig. 5. The coarse-scale soil moisture product showed similar R values to those of the XGBoost model developed using all stations, and it also yielded large bias and dispersed RMSE and ubRMSE values at most stations.

After filtering the stations using the TC method, the accuracies of the ERA5-Land soil moisture product at those representative stations improved significantly. Similarly, the validation accuracies of the model developed using the representative stations also improved significantly, with the distribution of its performance metrics being more concentrated across stations, compared to the model developed without station filtering. In particular, the median R of the model at each station increased from 0.64 to 0.74, median RMSE decreased from 0.075 to 0.068 m3 m-3, and ubRMSE decreased from 0.055 to 0.052 m3 m-3. Over most of the representative stations, the XGBoost model obtained similar or even larger R values compared to the ERA5-Land soil moisture product. However, there were also several stations where the model achieved relatively lower R values, yet this degradation in temporal metrics with respect to the original coarse-scale products can be found in many soil moisture downscaling studies (Gruber et al., 2020).

On the other hand, the model developed using the representative stations still exhibited a large bias at most stations, ranging from -0.21 to 0.21 m3 m-3, although the median bias of the model was 0. Therefore, the decreased overall accuracies of the model over unknown spaces can be attributed to these large site-specific biases, which may be caused by the high spatiotemporal variability of surface soil moisture and the scale differences between the target point-scale soil moisture and 1 km model predicted soil moisture. Specifically, in random and year-independent validation strategies, part of the site-specific information is known to the models; whereas in the site-independent validation method, this information is entirely unknown to the model. By adopting the TC method, it is possible to select soil moisture stations that are representative of the average soil moisture on a larger scale, thereby alleviating the scale difference issue to some extent. However, there may still be large biases between measurements from these point-scale representative soil moisture stations and footprint-scale average soil moisture values. As these biases are site-specific, can be positive or negative, and have a median value for all samples near 0, the overall ubRMSE that the model achieved on the site- or year-independent test samples can still be large when these biases are unknown to the model. Nevertheless, training the model with representative soil moisture stations not only improved the model's overall performance over unknown spatiotemporal locations (Table 4) but also improved the performance metrics of the model at each station (Fig. 5).

In addition to the performance metrics of the two XGBoost models at each station, Table 6 shows the validation accuracies of the model developed using the representative stations over different land cover types. Affected by a series of practical factors, the distribution of ISMN soil moisture stations is uneven in space, with the majority of the stations located in the CONUS. After screening stations via the TC method, the spatial distribution of representative stations remained uneven, with the resulting number of stations for each land cover type also varying significantly (Fig. 1). Overall, the performance of the model developed using the representative stations for most land cover types showed an improvement compared with the model developed using all stations, as indicated by larger median R values and smaller median RMSE and ubRMSE values. However, the median ubRMSE of the model achieved for forests was larger than that for other land cover types, likely a result of soil moisture maintaining at high levels in forested areas. Additionally, among the seven land cover types, the model achieved the lowest median R values for shrublands and barren lands, likely due to the limited number of stations present across these two types. However, the model also achieved the lowest median ubRMSE values for these two types, which can be partly attributed to the fact that despite the low sample percentages the number of samples for these land cover types was sufficient for the models to learn, as well as in part due to the relatively simple soil moisture dynamics of these two types. Although the median bias of the model for each land cover type was near 0, the model exhibited a large prediction bias for most stations across each land cover type (Table 6). After removing the prediction bias at each station, the median ubRMSE of the model for the seven land cover types ranged from 0.031 to 0.061 m3 m-3, marking a dramatic decrease over the corresponding median RMSE. Given that a large prediction bias existed in each land cover type and that the model performance did not vary significantly across different types, it was suggested that the uneven distribution of land cover types across samples was not the major cause of the decreased overall model accuracy over unknown spaces.

Using the XGBoost model developed above, a global 1 km spatiotemporally continuous soil moisture product (GLASS SM) was generated. To intuitively demonstrate the ability of this product for capturing the temporal variations in soil moisture over an unknown space, four independent networks under different climatic and environmental conditions were selected, and the time-series curves of the GLASS and measured soil moisture for these networks were compared. Considering the high spatiotemporal variability of surface soil moisture and the scale differences between point-scale observations and the 1 km GLASS SM product, the mean measured soil moisture curve was first calculated by averaging soil moisture curves from all stations within a network and then compared with the mean predicted soil moisture curve calculated using all corresponding pixels of the GLASS SM product within that network. Moreover, as an input variable of the 1 km soil moisture estimation model, the time-series curves of the ERA5-Land reanalysis soil moisture product over the four independent networks were also extracted as a reference.

In most cases, the GLASS soil moisture curves were much closer to the measured values than the time-series curves of the ERA5-Land reanalysis soil moisture product in both the YA and YB soil moisture networks (Fig. 6a, b). The R values between the GLASS and measured soil moisture for these two networks were 0.84 and 0.89, respectively, which were slightly higher than the ERA5-Land soil moisture (0.80 and 0.84); whereas the ubRMSE values were 0.048 and 0.034 m3 m-3, respectively, slightly lower than the ERA5-Land soil moisture product (0.052 and 0.044 m3 m-3). Accordingly, over these two relatively dense soil moisture networks, the 1 km GLASS SM product can basically capture the dynamics of measured soil moisture. However, underestimates occurred at some high-value intervals on the measured soil moisture curves, which may be caused by nearby irrigation at some stations within agricultural regions, where the GLASS SM product may not be able to capture such patterns, given that irrigation is usually not uniformly distributed in space.

For the Fort Cobb and Little Washita soil moisture networks, both the GLASS and ERA5-Land soil moisture estimates basically captured the dynamics of measured soil moisture (Fig. 6c, d). Specifically, the R values between the mean GLASS and measured soil moisture for these two networks were 0.69 and 0.76, respectively, slightly lower than the ERA5-Land soil moisture product (0.74 and 0.77). However, both the GLASS and ERA5-Land reanalysis soil moisture products showed a large positive bias throughout most of the observation period, particularly in the Little Washita network. This is likely because these two soil moisture networks cover a relatively large watershed containing only a few stations. Nevertheless, the ubRMSE values between the mean GLASS and measured soil moisture values for these two networks were 0.037 and 0.033 m3 m-3, respectively, which were significantly lower than those for the ERA5-Land soil moisture (0.047 and 0.046 m3 m-3). Overall, above results suggested that the derived product can accurately capture the temporal variations of in situ soil moisture under different climatic conditions. Further, the GLASS SM product achieved similar R values as the ERA5-Land product across these networks, with the R values ranging from 0.69 to 0.89 and the ubRMSE values ranging from 0.033 to 0.048 m3 m-3.

After producing the global 1 km spatiotemporally continuous GLASS SM product, it was compared with two global microwave soil moisture products for spatiotemporal consistency. The first product selected for comparison was SPL2SMAP_S, the first publicly released global soil moisture product at a spatial resolution of 1 km. Because the SPL2SMAP_S 1 km product has a temporal resolution of 12 d over most global areas and as it has many spatial gaps at the daily scale, spatial synthesis of the SPL2SMAP_S dataset was conducted during a 12 d period with relatively high spatial coverage before comparison. Figure 7 shows the spatial distribution of the SPL2SMAP_S 1 km soil moisture product, synthesized from 3 to 15 October 2016, alongside the 1 km spatiotemporally continuous GLASS SM map for 9 October 2016. Here, it can be seen that the 12 d synthetic SPL2SMAP_S soil moisture product still has large spatial gaps (e.g., the western continental United States, western China, and southwestern Australia), whereas the GLASS SM product has a substantially more complete spatial coverage (except for the high-latitude regions during the cold seasons). With regards to the spatial distribution characteristics, both soil moisture products with 1 km resolutions exhibits a high level of consistency, with higher soil moisture levels found in the tropics, eastern USA, and southeastern China and with lower levels observed in deserts (e.g., Sahara) and other semiarid regions.

To quantitatively investigate the spatial consistency between these two 1 km soil moisture products, spatial R and RMSD between them were calculated for each 12 d of 2016 using collocated pixels, after removing soil moisture estimates larger than 0.6 m3 m-3 from the SPL2SMAP_S product. As displayed in Fig. 8a, the spatial R (orange line) between the GLASS and SPL2SMAP_S products ranges from 0.61 to 0.67, with a median value of 0.62, partially affected by the discontinuous spatial coverage of the SPL2SMAP_S product. The spatial RMSD (orange dots) between the two 1 km products in 2016 ranges from 0.098 to 0.106 m3 m-3, and the relatively large RMSD values may be attributed to the greater spatial heterogeneity (e.g., terrain and soil texture) at fine scales which could cause large disparities in soil moisture estimates from different algorithms. Overall, both qualitative and quantitative comparisons suggested a good and stable spatial consistency between the 1 km GLASS and SPL2SMAP_S microwave soil moisture products.

The second global product selected for comparison was the widely used ESA CCI combined soil moisture dataset with a spatial resolution of 0.25∘. Because the CCI soil moisture product has a daily temporal resolution and more complete spatial coverage, more quantitative analyses can be conducted when comparing with the 1 km spatiotemporally continuous GLASS SM product. Figure 9 shows the spatial distribution of the CCI active–passive microwave combined soil moisture and GLASS SM resampled to 0.25∘ for 4 d from different seasons in 2016, as well as the corresponding scatterplots of these two soil moisture products. The high spatial consistency between the CCI soil moisture product and resampled GLASS SM product on different dates is readily apparent, as both products display lower soil moisture values in arid regions, including the western USA, northern and southern Africa, the Middle East, central and western Asia, and Austria, and higher soil moisture values in tropical and temperate regions, such as central Africa, southern Asia, the eastern USA, and southeastern China. Although CCI estimates incorporate a variety of active and passive microwave soil moisture products, its spatial coverage remains incomplete, partly due to observation gaps of the sensors and the physical limitations of microwave soil moisture retrieval algorithms (Dorigo et al., 2017), such as failing to provide accurate soil moisture predictions on densely vegetated land surfaces (e.g., the Amazon River and Congo basins). In contrast, the GLASS SM product shows greater spatial integrity, except at high latitudes in cold seasons due to low temperatures and frozen soils.

As shown in Fig. 8b, the daily spatial R between the resampled GLASS and ESA CCI soil moisture products at 0.25∘ resolution in 2016 ranges from 0.72 to 0.86, with a median value of 0.82, indicating that the two products exhibit high spatial consistency across the seasons. As a comparison, the spatial R and RMSD between the CCI and two other resampled soil moisture products (ERA5-Land and SPL2SMAP_S) were also calculated and plotted. It is clear that the spatial R curves of the resampled ERA5-Land (blue) and GLASS (orange) at 0.25∘ only differ slightly, which is to be expected given that the coarse-scale ERA5-Land soil moisture was used to provide background soil moisture information for our model and given that it achieved the highest importance score among all the input variables. Both curves exhibit significant seasonal variation, with higher spatial R values in spring and winter than in summer or autumn, possibly related to the larger differences between the two resampled products (GLASS and ERA5-Land) and CCI over high latitudes. However, the spatial RMSD curves of the ERA5-Land and GLASS differ significantly. While the blue dotted line (RMSD between CCI and ERA5-Land) exhibits an opposite seasonal pattern to the R curves, with RMSD ranging widely from 0.086 to 0.12 m3 m-3, the orange dotted line (RMSD between CCI and GLASS) is more stable, with RMSD ranging from 0.068 to 0.087 m3 m-3. Besides, as also shown in Fig. 8b, although the resampled SPL2SMAP_S soil moisture product has the most stable spatial R and RMSD curves (gray), it achieves relatively lower spatial R values and larger spatial RMSD values than those of the resampled GLASS product at 0.25∘, suggesting its relatively lower level of spatial consistency with the CCI product. This is to our surprise considering that both the SPL2SMAP_S and CCI soil moisture products were derived from microwave satellite observations, and a possible cause for this could be the discontinuous spatial coverage of the SPL2SMAP_S product.

Note that the GLASS SM product displays a general underestimation relative to the CCI combined soil moisture (Fig. 9i–l). Although the overestimation of the CCI soil moisture product has been reported in a previous study, particularly for equatorial (savanna) regions (Al-Yaari et al., 2019), the GLASS SM product may also contain some biases, which jointly contribute to the RMSD between them. Figure 10 shows a zoomed-in comparison between the 1 km GLASS and 0.25∘ ESA CCI microwave soil moisture product in western China on 28 June 2016, with the corresponding 0.1∘ ERA5-Land reanalysis soil moisture product, which is one of the main inputs to the XGBoost model, also shown as a reference. In general, the GLASS product exhibits spatial consistency with both coarse-scale soil moisture products, with lower soil moisture levels in the Junggar Basin, Tarim Basin, Qaidam Basin, and western part of the Tibetan Plateau, and higher soil moisture levels in the Tianshan Mountains, Ili River valley, and southeastern part of the plateau where the vegetation is also much denser. Specifically, in the southeastern Tibetan Plateau, the GLASS and CCI soil moisture products show higher consistency, while the ERA5-Land soil moisture product is suspected to be underestimated. Moreover, it is clear that the 1 km GLASS SM product is not only spatially complete but also contains more spatial details which can well reflect the distribution patterns of terrain and vegetation.

In addition to the spatial consistency analysis described above, the temporal consistency between the CCI and the spatiotemporally continuous GLASS SM product was also explored. Specifically, for each pixel of these two products with > 30 d of concurrent predictions, the R and RMSD between the time-series soil moisture predictions were calculated separately for 2016, and the spatial distribution of these two metrics is shown in Fig. 11. The correlation between the two products was high in most areas, except the Sahara, high latitudes, and some localized regions. The relatively low or even negative R values between the two products in the Sahara is likely due to the fact that soil moisture in this region is close to zero, and a small difference in temporal variation may lead to poor correlation. It can also be seen from Fig. 11b that the RMSD values between the two products in the Sahara were rather small. The relatively low R values between the two products at high latitudes may be attributed to the irregular prediction frequency of the CCI product at high latitudes and the rapid change in soil moisture during the freeze–thaw transition period in this region, which possibly cause larger errors in both products and thus increased temporal inconsistency. Greater differences between soil moisture products at high latitudes have also been found elsewhere (Wang et al., 2021). Further, no obvious patterns were revealed regarding the distribution of RMSD between the two soil moisture products, as the regions with relatively large RMSD values were rather scattered.