LGHAP v2: a global gap-free aerosol optical depth and PM_2.5 concentration dataset since 2000 derived via big Earth data analytics

The Long-term Gap-free High-resolution Air Pollutants (LGHAP) concentration dataset generated in our previous study has provided spatially contiguous daily aerosol optical depth (AOD) and fine particulate matter (PM_2.5) concentrations at a 1 km grid resolution in China since 2000. This advancement empowered unprecedented assessments of regional aerosol variations and their influence on the environment, health, and climate over the past 20 years. However, there is a need to enhance such a high-quality AOD and PM_2.5 concentration dataset with new robust features and extended spatial coverage. In this study, we present version 2 of a global-scale LGHAP dataset (LGHAP v2), which was generated using improved big Earth data analytics via a seamless integration of versatile data science, pattern recognition, and machine learning methods. Specifically, multimodal AODs and air quality measurements acquired from relevant satellites, ground monitoring stations, and numerical models were harmonized by harnessing the capability of random-forest-based data-driven models. Subsequently, an improved tensor-flow-based AOD reconstruction algorithm was developed to weave the harmonized multisource AOD products together for filling data gaps in Multi-Angle Implementation of Atmospheric Correction (MAIAC) AOD retrievals from Terra. The results of the ablation experiments demonstrated better performance of the improved tensor-flow-based gap-filling method in terms of both convergence speed and data accuracy. Ground-based validation results indicated good data accuracy of this global gap-free AOD dataset, with a correlation coefficient (R) of 0.85 and a root mean square error (RMSE) of 0.14 compared to the worldwide AOD observations from the AErosol RObotic NETwork (AERONET), outperforming the purely reconstructed AODs (R = 0.83, RMSE = 0.15), but they were slightly worse than raw MAIAC AOD retrievals (R = 0.88, RMSE = 0.11). For PM_2.5 concentration mapping, a novel deep-learning approach, termed the SCene-Aware ensemble learning Graph ATtention network (SCAGAT), was hereby applied. While accounting for the scene representativeness of data-driven models across regions, the SCAGAT algorithm performed better during spatial extrapolation, largely reducing modeling biases over regions with limited and/or even absent in situ PM_2.5 concentration measurements. The validation results indicated that the gap-free PM_2.5 concentration estimates exhibit higher prediction accuracies, with an R of 0.95 and an RMSE of 5.7 µg m⁻³, compared to PM_2.5 concentration measurements obtained from former holdout sites worldwide. Overall, while leveraging state-of-the-art methods in data science and artificial intelligence, a quality-enhanced LGHAP v2 dataset was generated through big Earth data analytics by cohesively weaving together multimodal AODs and air quality measurements from diverse sources. The gap-free, high-resolution, and global coverage merits render the LGHAP v2 dataset an invaluable database for advancing aerosol- and haze-related studies as well as triggering multidisciplinary applications for environmental management, health-risk assessment, and climate change attribution. All gap-free AOD and PM_2.5 concentration grids in the LGHAP v2 dataset, as well as the data user guide and relevant visualization codes, are publicly accessible at https://zenodo.org/communities/ecnu_lghap (last access: 3 April 2024, Bai and Li, 2023a).

1 Introduction

Atmospheric aerosols, produced from either natural or anthropogenic emissions, have been proven to pose significant threats to human health, the ambient environment, and climate (Up in the aerosol, 2022). The risks to public health from aerosol pollution are evident, with about 4.2 million deaths per year attributable to the exposure of fine aerosol particles, as stated by the World Health Organization (WHO, 2022). With increased aerosol loading, aerosols can significantly impair atmospheric visibility because of the hygroscopic effect, thereby reducing direct solar radiation on Earth's surface (Liu et al., 2020; Wang and Yang, 2014; Wild et al., 2021; Yang et al., 2016). In addition to the evident influence on air quality (Li et al., 2017), atmospheric aerosols have an important and complex influence on regional and even global climate (Guo et al., 2016, 2019; Li et al., 2019; Yang et al., 2020; Zhao et al., 2020). Therefore, accurate monitoring of the atmospheric aerosol loading is vital for improving our understanding of the human-driven ambient environment and exposure pathways in health-risk assessment.

Aerosol optical depth (AOD), a measure of aerosols distributed within an air column from Earth's surface to the top of the atmosphere, has been widely used as a key indicator of total atmospheric aerosol loading. Ground-based aerosol observing networks, e.g., the internationally collaborated AErosol RObotic NETwork (AERONET), China Aerosol Remote Sensing Network (CARSNET), and Sun-Sky Radiometer Observation Network (SONET), have long served as the ground truth for AOD monitoring (Che et al., 2015; Giles et al., 2019; Li et al., 2018). However, the sparse distribution of aerosol monitoring stations poses a significant challenge in gaining a comprehensive understanding of the aerosol variations across the globe.

Satellite-based AOD data bridge this gap by providing spatially resolved AOD retrievals with extensive spatial coverage. Over the past 40 years, a variety of spaceborne instruments, e.g., the Sea-Viewing Wide Field-of-View Sensor (SeaWiFS), Moderate Resolution Imaging Spectroradiometer (MODIS), Visible Infrared Imaging Radiometer Suite (VIIRS), and Polarization and Directionality of the Earth's Reflectances (POLDER), were deployed on board various satellite platforms and launched into space (Wei et al., 2020). These versatile instruments provide ample AOD and aerosol property measurements, enabling us to map the global AOD distribution with finer spatial resolutions. Nonetheless, satellite-based AOD retrievals often suffer from excessive data gaps because of extensive cloud cover and retrieval failures, significantly impairing the data application potential and resulting in large uncertainties when assessing the influence of aerosol on weather and climate.

A variety of gap-filling methods were developed and applied to reconstruct the missing values in the remotely sensed satellite AOD images (Wei et al., 2020; Xiao et al., 2021). The simplest method is to fill in data gaps with valid observations from alternative data sources, e.g., filling in data gaps in MODIS AOD images from Terra with AOD observations from Aqua (Bai et al., 2019; Sogacheva et al., 2020) or fusing with AOD simulation outputs from numerical models (Xiao et al., 2021). Such a substitution method is straightforward and effective, particularly in an era with big Earth observation data. Nonetheless, cross-mission biases are always salient between satellite-based retrievals because of the significant differences in instrument properties and/or retrieval algorithms. Thus, bias correction is essential for reducing systematic biases (Bai et al., 2016b, a), and methods such as linear regression and maximum likelihood estimation are often applied for this purpose (Bai et al., 2016a, b, 2019; Ma et al., 2016; Xu et al., 2015). More complex methods, like Bayesian maximum entropy, were also applied to fuse AOD products, even with varying spatial resolutions (Tang et al., 2016; X. Wei et al., 2021).

Another type of gap-filling method works, in principle, to recover missing information via dominant pattern recognition and reconstruction over space and time, and the Data INterpolating Empirical Orthogonal Functions (DINEOF) method is a representative one (Beckers and Rixen, 2003; Liu and Wang, 2019). Two similar methods were developed to fill data gaps in the ground-measured particulate matter (PM_2.5) concentration time series and geostationary satellite-sensed AOD images (Bai et al., 2020; L. Li et al., 2022). Similarly, Zhang et al. (2022) developed a spatiotemporal fitting algorithm to fill gaps in the daily MODIS AOD product by predicting AOD values based on annual trends and spatial residues inferred from neighboring pixels. Nonetheless, filling data gaps with a single data source is always challenging, particularly for those with extensive missing values (e.g., satellite-based AOD). Learning missing values from external information, e.g., numerical AOD simulations (Li et al., 2020; Xiao et al., 2017) and meteorological factors (Bi et al., 2019), was proven an effective and feasible way of improving the spatial coverage of reconstructed AOD fields.

The tensor-flow-based method, a more complex big data analytics framework, was developed to integrate six satellite-based AOD datasets, numerical aerosol diagnostics, and in situ air quality measurements, while a machine learning method, i.e., random forest, was applied for downscaling and bias-correction purposes (Bai et al., 2022a). Harnessing multimodal data fusion and missing value reconstruction capabilities, a long-term gap-free high-resolution MODIS-like AOD dataset (LGHAP v1) was successfully generated in China, with an overall data accuracy comparable to raw satellite retrievals, from which gap-free PM_2.5 and PM₁₀ concentrations were mapped on a daily basis. Despite the good performance, additional investigations have recently proven the critical importance of prior information for tensor-flow-based gap-filling, particularly over areas with substantial missing values (Bai et al., 2022a; K. Li et al., 2022; L. Li et al., 2022). Moreover, the strategies of maintaining an invariant background field and assigning equal weights to different AOD inputs may slow down the convergence speed and degrade the reconstruction accuracy.

In this study, we present a new global-scale LGHAP dataset, referred to as LGHAP v2 hereafter, which extends daily gap-free AOD and PM_2.5 concentrations from China to worldwide at a 1 km grid resolution for the period of 2000–2021. To accommodate massive global Earth observations acquired from diverse sources, an improved big Earth data analytics approach was developed by harnessing several new algorithmic improvements to enhance the tensor-flow-based AOD gap-filling. Moreover, a novel deep-learning method, i.e., the SCene-Aware ensemble learning Graph ATtention network (SCAGAT), was applied to fulfill far more accurate PM_2.5 concentration mapping across the globe, particularly over regions with limited air quality monitoring stations. Benefiting from the customized algorithmic improvements and the innovative SCAGAT PM_2.5 concentration mapping approach, the LGHAP v2 dataset has not only an extended spatial coverage from China to worldwide, but also improved data accuracy. As a publicly accessible and global long-term gap-free MODIS-like AOD and PM_2.5 concentration dataset, LGHAP v2 serves as a promising data source to improve our understanding of global aerosol pollution dynamics and its adverse impacts on public health, ecosystems, weather, and climate.

2 Data sources

Similarly to our previous study, here we aim to synergistically integrate the big Earth data acquired from diverse sources to generate a global long-term gap-free AOD dataset with a daily 1 km resolution, from which spatially contiguous PM_2.5 concentration estimates can then be derived using a more robust and accurate data-driven approach. Table 1 describes the array of big Earth data employed in this study, including gridded AOD products from six polar-orbiting satellites, numerically simulated Modern-Era Retrospective Analysis for Research and Applications version 2 (MERRA-2) aerosol diagnostics, 10 meteorological reanalysis fields, and datasets of in situ AOD and air pollutant concentration measurements. Additionally, auxiliary parameters representing land use and land cover types, elevation, population density, and vegetation covers were also employed as critical explanatory variables to harmonize discrepancies among multimodal heterogeneous aerosol datasets. Note that the spatial and temporal resolution as well as the time period for each data product are different from those of the benchmark dataset, i.e., the Multi-Angle Implementation of Atmospheric Correction (MAIAC) AOD product, and a data homogenization method is therefore essential to account for such discrepancies to reduce possible bias propagation in the subsequent data fusion procedure.

Table 1 Summary of the diverse big Earth data used in this study to generate global gap-free AOD and PM_2.5 concentrations at daily and 1 km resolutions (LGHAP v2) from 2000 to 2021. Note spatial resolution is not applicable (n/a) for site-based data.

2.1 Satellite-based AOD products

The AOD retrievals, derived from the MODIS sensor on board Terra using the MAIAC algorithm (denoted as AOD_Terra afterwards), were hereby used as the benchmark for generating the global long-term gap-free AOD dataset, given their finer spatiotemporal resolution and longer temporal coverage (Lyapustin et al., 2011, 2018; Mhawish et al., 2019). Previous studies have demonstrated the superior quality of AOD_Terra relative to other gridded AOD products (Chen et al., 2021; Martins et al., 2017; Qin et al., 2021) in regard to data accuracy and spatiotemporal completeness, even better than those retrieved with the well-known Dark Target and Deep Blue algorithms (Jiang et al., 2023; Liu et al., 2019). Figure S1 in the Supplement presents the spatial and temporal distribution of the coverage ratio of valid AOD_Terra from 2000 to 2021 at each satellite footprint across the globe.

Satellite-based AOD retrievals from a few key instruments other than MODIS were also applied to support gap-filling of AOD_Terra, and they include the (1) VIIRS on board Suomi-NPP, (2) Multi-angle Imaging SpectroRadiometer (MISR, on board Terra), (3) Advanced Along-Track Scanning Radiometer (AATSR, on board Envisat), (4) POLDER on board PARASOL, and (5) SeaWIFS on board SeaStar. Meanwhile, MAIAC AOD data from MODIS on board Aqua were also applied as an important complementary data source. Given their varied overpassing times and temporal spans, these multisensory AOD datasets can provide complementary observations to help reduce random errors during the AOD data reconstruction procedure because of the known prior knowledge. More details of these AOD products can be found in Bai et al. (2022a) and Wei et al. (2020).

2.2 Ground-based AOD observations and air quality measurements

2.2.1 AERONET AOD observations

Ground-based AOD observations from AERONET have long been used as the ground truth for validating AOD retrievals from other instruments, particularly diverse satellite-based AOD retrievals. In this study, AOD observations from AERONET during the study period were employed as an independent data source to validate the data accuracy of the global gap-filled AOD dataset. To guarantee an adequate number of AERONET AOD samples, the Level 1.5 AOD observations instead of Level 2.0 were applied, though the latter has stricter screening criteria for quality control. For spatial registration, each AERONET AOD observation was spatially collocated with mean AOD values over grids within a 5 × 5 km window size. Figure S2 presents the spatial distribution of the AERONET sites used in this study.

2.2.2 Air quality measurements

Concentrations of PM_2.5 and other relevant air pollutants, like NO₂, SO₂, PM₁₀, and CO, were acquired from a few environmental agencies and monitoring centers, e.g., the United States Environmental Protection Agency, European Air Quality Portal, China National Environmental Monitoring Centre, Canada National Air Pollution Surveillance, and Japan National Institute for Environmental Studies, to name a few. Moreover, air quality measurements acquired from the World's Air Pollution Index, an open-source data hub, were included as well. Given potential differences in measuring principles and quality control criteria, we performed rigorous data cleaning measures to harmonize these multisource air quality measurements, including not only the removal of outliers, but also a unification of timescales to the daily average. Aiming to provide critical information to facilitate the AOD gap-filling, ground-based air quality measurements were used as an important proxy for regional in situ AOD prediction, largely because of the relatively dense distribution of air quality monitoring networks and the associations between aerosol loadings and regional air pollutant concentrations.

Atmospheric visibility, a common air quality indicator highly associated with aerosol loadings, was acquired from worldwide meteorological monitoring stations and used to predict AOD over each monitoring site via data-driven modeling. Given the much denser distribution of ambient air quality and meteorological monitoring sites, as shown in Fig. S2 in the Supplement, a global virtual AOD monitoring network was in turn established, harnessing the associations between AOD and air-quality-relevant parameters. Such a virtual network provides us with an unparalleled opportunity to improve AOD gap-filling accuracy and efficiency, particularly over regions with massive data voids in satellite AOD images (Bai et al., 2022b; Li et al., 2022b).

2.3 Numerical simulations

2.3.1 MERRA-2 aerosol diagnostics

The MERRA-2 aerosol diagnostics, including total AOD and aerosol components like black carbon, organic carbon, dust, and sulfate aerosols, were employed to provide prior information to advance AOD gap-filling. As NASA's latest reanalysis for the satellite era, MERRA-2 is generated using the new Earth system model, the Goddard Earth Observing System version 5 (GEOS-5), providing global simulations of a variety of geophysical and chemical variables on Earth's surface. More details of the assimilation system and the data quality of MERRA-2 aerosol reanalysis can be found in Buchard et al. (2017) and Randles et al. (2017). By taking AOD_Terra as the learning target, data-driven models were established to spatially downscale and bias-correct the MERRA-2 AOD field, with meteorological, geographical, and socioeconomic factors used as covariates. This downscaled and bias-corrected MERRA-2 AOD field, given its spatially contiguous coverage, was then used as critical information to facilitate the gap-filling of AOD_Terra.

2.3.2 ERA5 reanalysis

As the latest atmospheric reanalysis produced by the European Centre for Medium-Range Weather Forecasts, ERA5 provides hourly estimates of a variety of atmospheric, terrestrial, oceanic, climatic, and meteorological variables. The data are provided for a 30 km grid resolution on Earth's surface, delineating the atmosphere layer using 137 levels from the surface up to a height of 80 km, covering the period from January 1940 to the present (Hersbach et al., 2020). Atmospheric parameters, including surface pressure, air temperature, relative humidity, wind speed, total column water, total precipitation, surface solar radiation downward, instantaneous moisture flux, and boundary layer height, were acquired from ERA5 and used as important modeling covariates in both data harmonization and PM_2.5 mapping models. A simple bilinear interpolation was applied to the ERA5 reanalysis data to convert them to the AOD_Terra footprint resolution for spatial registration.

2.4 Auxiliary data

Several socioeconomic and geographic factors were also applied as covariates to support AOD gap-filling and PM_2.5 concentration mapping. Specifically, gridded population data from WorldPop were used to indicate the spatial distribution of residents, serving as a critical proxy for anthropogenic air pollutant emission intensity. To characterize the land-use-dependent aerosol emissions, land cover types and the vegetation index derived from MODIS products along with the coverage ratio of an impervious surface calculated from the land use dataset generated by Huang et al. (2022) were also applied. The digital elevation data collected from the Shuttle Radar Topography Mission (SRTM) with a resolution of 1 arcsec were used to characterize the potential impact of topography on aerosol loadings.

3 Methods

3.1 Tensor-flow-based AOD reconstruction

3.1.1 Overview of the AOD gap-filling method

Deriving spatially contiguous PM_2.5 concentrations from gap-filled AOD images has proven more promising for a better analysis of large-scale PM_2.5 distribution (Bai et al., 2022b). In this study, the big Earth data analytics framework proposed in Bai et al. (2022a) was further adapted and improved for generating global gap-free AOD images to support various content-based mappings. As shown in Fig. 1, the improved big Earth data analytics framework also consists of three primary data manipulation procedures, including (1) machine-learned multimodal data homogenization, (2) knowledge-reinforced AOD tensor compiling, and (3) tensor-flow-based AOD reconstruction, with algorithmic improvements primarily conducted in the latter two procedures. This improved big Earth data analytics approach empowered us to weave together multimodal AODs and versatile big Earth observations from diverse sources via a synergy of state-of-the-art machine learning and tensor completion methods. Because the technical flow of this big Earth data analytics framework was previously detailed in Bai et al. (2022b), here we only provide an overview of this method while describing more details of the newly developed algorithmic components in the following subsections.

Figure 1 A schematic illustration of the improved big Earth data analytics for generating the MODIS-like global gap-free AOD dataset.

The overall architecture of this big Earth data analytics framework was summarized as follows. Multimodal AODs and relevant aerosol data acquired from different satellites, ground monitoring stations, and numerical models were first harmonized to resemble the baseline dataset of AOD_Terra, aiming to minimize both cross-sensor biases and spatial heterogeneities. This data homogenization process is vital for the tensor-flow-based AOD gap-filling, because the bias-corrected and downscaled AOD estimates were critical inputs to form the AOD data cube. More details related to the multisource data homogenization are given in Sect. S1 in the Supplement. To fill data gaps in each individual AOD_Terra image, an AOD data cube was then constructed by aggregating harmonized multisensory AOD data on the same date along with historical AOD_Terra images resembling similar spatial patterns over the same region. Because of the excessive nonrandom missing values in the AOD_Terra images, both the downscaled MERRA-2 AOD grids and AOD estimates derived from air quality and visibility measurements were used conjunctively to identify similar AOD_Terra images from the historical image series. The selected historical AOD_Terra images and bias-corrected AOD images from other satellites on the same date were used individually as a slice of the tensor. Additionally, dispersed in situ AOD estimates and 5 % of the randomly selected downscaled MERRA-2 AOD data were directly overlaid onto the corresponding AOD_Terra grids without valid AOD retrievals. These implementations helped improve the gap-filling accuracy and greatly boosted the convergence speed given the provision of prior knowledge.

High-order singular value decomposition (HOSVD), an orthogonal Tucker decomposition method, was applied to each well-compiled AOD data cube for tensor-flow-based pattern recognition and data completion. Data gaps within the input AOD tensor were first filled with the spatial average of each individual AOD image to initialize the tensor decomposition. The AOD tensor was then decomposed along each two-dimensional slice independently, and a new tensor was subsequently reconstructed based on the principal modes via a low-rank approximation (i.e., generating an approximating matrix with reduced rank for compression). During this procedure, the AOD_Terra observations in the target image to be gap-filled were deemed hard data (i.e., true-state and invariant throughout the tensor completion procedure), while multisensory AOD estimates and historical AOD_Terra images served as soft data (Supplement and updated by iterates till convergence). By iteratively adjusting the dimension-varied ranks, the data values over grids to be gap-filled were updated and tuned to optimize both spatial homogeneity and information entropy concurrently (Bai et al., 2020, 2022a). The tensor completion process continued till it reached an agreement (with a bias decay ratio < 0.1 %) between the reconstructed values and the previously reserved AOD_Terra observations.

3.1.2 Algorithmic improvements

To accommodate the massive data analytics for global-scale AOD gap-filling, three major algorithmic enhancement modules were incorporated to help improve reconstruction efficiency and accuracy, with particular focus on the optimization of data manipulation procedures in tensor-flow-based AOD gap-filling. Algorithm 1 presents the pseudo code of the optimized algorithm used for tensor-flow-based AOD reconstruction.

Algorithm 1 The pseudo code of the optimized algorithm used for tensor-flow-based AOD reconstruction.

Attention-reinforced AOD tensor construction

In our previous study, both the target data (i.e., the AOD_Terra image) and the soft data (i.e., AOD estimates from other data sources and historical AOD_Terra images) were treated equally in the AOD tensor throughout the tensor decomposition and reconstruction process (Bai et al., 2022a). This indifferent data treatment strategy neglected the information abundance of soft data and the spatial similarity between the soft and target data, making the reconstructed field more likely to resemble the dominant patterns learned from images with fewer data gaps rather than those with spatial patterns similar to the target image. To account for this drawback, an attention mechanism was hereby introduced to assign different weights to each data slice in the input AOD tensor, aiming to improve the AOD reconstruction performance by learning from spatiotemporal features embedded in more relevant data fields instead of all the available data.

As a widely used technique in deep learning, the attention mechanism is a mimic of cognitive attention allowing the model to focus on specific parts of the input data, achieved by assigning higher weights to more crucial elements in ensemble learning. Regarding the tensor-flow-based AOD reconstruction task, data slices with a greater similarity to the target image and fewer data gaps are supposed to play more important roles than less similar ones with extensive data gaps during tensor completion. Three statistical metrics, including mutual information (Shannon, 1948), the spatial coverage ratio of common observations (R_common) between soft data and hard data, and the spatial coverage ratio of extra observations beyond common observations in soft data (R_extra), were calculated to determine the overall weight that should be assigned to each slice of data in the input AOD tensor. Specifically, mutual information was applied to characterize the mutual dependence between the target image and each slice of soft data, while the common spatial coverage ratio was used to indicate the data amount for mutual information calculation, and the extra spatial coverage ratio was employed to depict additional information content that can be provided by soft data. Equations (1)–(3) provide the formulas to calculate these three statistical metrics.

$$\begin{array}{}\text{(1)}& {\displaystyle }\mathrm{MI}(X,Y)=\sum _{y\in Y}\sum _{x\in X}p\left(x,y\right)\log \left({\displaystyle \frac{p\left(x,y\right)}{p\left(x\right)p\left(y\right)}}\right)\text{(2)}& {\displaystyle }{R}_{\mathrm{common}}=\mathrm{\Phi }(X,Y)\times \mathrm{100}\mathit{\%}\text{(3)}& {\displaystyle }{R}_{\mathrm{extra}}=\mathrm{\Phi }\left(\stackrel{\mathrm{̃}}{X},Y\right)\times \mathrm{100}\mathit{\%}\end{array}$$

Note that X and Y refer to common observations in soft and hard data, respectively. $\stackrel{\mathrm{̃}}{X}$ denotes extra observations in soft data. p(x,y) is the joint probability mass function of X and Y, while p(x) and p(y) are the marginal distribution mass functions of X and Y, respectively. Additionally, Φ(X,Y) is the spatial coverage ratio of the common observations, and $\mathrm{\Phi }\left(\stackrel{\mathrm{̃}}{X},Y\right)$ is the spatial coverage ratio of extra observations in the soft data. By multiplying these three normalized weights by the corresponding soft data, an attention-reinforced AOD tensor was constructed in turn, which was then used as the input data cube for tensor completion.

Adaptive prior information updating

To facilitate the AOD gap-filling over regions with substantial data gaps, in our previous method, 5 % random samples from the downscaled MERRA-2 AOD image (AOD_M2 hereafter) on the same date were used as prior information and directly overlaid onto grids without observational AOD (i.e., AOD_Terra and site-based AOD estimates from air quality and visibility measurements). Although this enabled us to improve the convergence speed during tensor completion, the spatial patterns of the reconstructed field over regions with excessive data gaps were more likely to resemble the distribution of AOD_M2 because of this unchanged prior information. In this context, large modeling biases in AOD_M2 might be introduced into the final reconstruction fields.

In this study, we introduced an adaptive prior information updating scheme to mitigate the potential bias propagation problem. The main principle is to force the AOD prior information in the input AOD tensor to update iteratively throughout the tensor completion process rather than maintaining it as invariant observations. Specifically, random AOD_M2 samples were only used to initialize the tensor construction, while weighted averages of the prior information and the corresponding reconstructed values were then used as new prior information for the next iteration. Meanwhile, the weights assigned to the reconstructed fields were gradually increased by iteration till convergence. The goal was to improve the contribution of reconstruction fields learning from actual observations while reducing the influence of background fields. The ablation experiments demonstrated the effectiveness of this scheme in improving the reconstruction performance over regions with limited observational data.

Optimized global data tile partition and rank updating

The high spatiotemporal resolution of AOD_Terra images presents a great challenge in performing global-scale AOD gap-filling because of the huge computational burden. To improve computational efficiency and to make the computing workload manageable, the following algorithmic adjustments were implemented. First, the continental AOD_Terra data worldwide were divided into 480 data tiles, with AOD gap-filling performed over each tile independently. Through a set of gap-filling trials with varying tile sizes, a nominal tile size covering 700 × 700 pixels (refer to Fig. S3 for the spatial distribution of the optimized data tiles) was finally applied to balance the computing workload and reconstruction accuracy. Moreover, a 50-pixel overlap on the boundary of each tile was enforced, and an inverse distance weighting scheme was applied to these overlapped pixels when mosaicking the gap-filled tiles, aiming to eliminate the boundary effects between tiles toward a smooth distribution of AOD across the globe.

Since the tensor's decomposition and reconstruction processes in the tensor completion are driven by iteratively updated tensor ranks, an optimized rank updating strategy was hereby proposed to improve the learning efficiency. Specifically, the ranks were updated in ascending order along with the first and second dimensions in the inner loops to enhance the spatial details of the reconstructed AOD fields. In contrast, the ranks were updated in a descending fashion along the third dimension in the outer loop to aggregate the target AOD_Terra image with the soft data in a low-rank approximation manner. This new rank updating strategy not only helps better resolve spatial details of AOD, but also accelerates the convergence speed of tensor completion.

3.2 Global PM_2.5 concentration modeling

The sparse and uneven distribution of ground-based air quality monitoring stations poses significant challenges to global PM_2.5 concentration mapping, particularly over regions with fewer PM_2.5 concentration measurements (e.g., Africa and South America in Fig. S2). Nonetheless, how to reinforce the spatial representativeness of data-driven models to improve the spatial extrapolation accuracy is still elusive. In this study, a recently developed deep-learning method, i.e., SCAGAT, was hereby applied to better estimate global PM_2.5 concentrations from gap-filled AOD images. Instead of establishing a single PM_2.5 estimation model using all available data samples collected from worldwide monitoring stations, site-specific PM_2.5 estimation models were first developed using random forest over each air quality monitoring station with adequate PM_2.5 concentration measurements.

Figure 2 Performance evaluation of different algorithmic enhancement modules on the reconstructed AOD distribution. Raw AOD_Terra denotes the actual AOD retrievals from Terra, while simulated AOD_Terra refers to partially masked AOD_Terra. The benchmark method is the AOD gap-filling approach proposed in Bai et al. (2022a). The latter three columns present the reconstructed fields using the enhanced benchmark methods. The R and bias denote the correlation coefficient and deviations between the holdout observed and reconstructed AOD data, respectively. The percent numbers shown in the two left panels indicate a spatial coverage ratio of valid AOD retrievals over the selected scenes.

For a given grid, raw PM_2.5 concentration estimates were estimated from a set of independent site-specific PM_2.5 estimation models, which should resemble similar geographic scene features to the given grid cell – under the assumption that the relationship between AOD and PM_2.5 is similar over regions with an analogous environmental background. Nine distinct factors covering geographic locations, land cover types, climate zones, AOD levels, and population density were utilized to characterize the scene attributes of each grid cell. Subsequently, a graph attention network was used to aggregate raw PM_2.5 concentration estimates derived from site-specific models to produce an ensemble estimate over the target grid cell. In the graph network, weights assigned to the adjacency matrix were determined in reference to the differences between nine different scene features, and the node bias was given as the testing accuracy of each site-specific PM_2.5 prediction model. This innovative ensemble learning method enables us to better predict PM_2.5 concentrations across the globe, particularly over regions with limited or even no in situ PM_2.5 concentration measurements. Figure S4 depicts the workflow of the proposed SCAGAT model, and additional details are introduced in Sect. S2. For more detailed descriptions of this method, please refer to Li et al. (2024).

Figure 3 Impact of the missing rate on the AOD gap-filling accuracy. The numbers on the top indicate the percentage of removed AOD data in the raw AOD_Terra image. The second row shows the distribution of the gap-filled AOD with zoomed-in maps present in the third row. The bottom panel presents scatter plots between the observed and reconstructed AOD.

4 Results

4.1 Efficacy assessment of algorithmic enhancement modules

Ablation experiments were first conducted to evaluate the accuracy improvement potential of each newly developed algorithmic enhancement module. Three case studies were simulated by masking actual AOD_Terra retrievals with randomly selected cloud masks on different dates, and the methods reinforced with different enhancement modules were then applied to reconstruct the former holdout AOD values. For intercomparison, the AOD gap-filling framework developed in Bai et al. (2022a) was used as the benchmark method. As shown in Fig. 2, the AOD distributions reconstructed using methods embedding attention mechanism and adaptive background information updating modules have smaller bias levels compared to the benchmark method, which in turn justifies the efficacy of these two new algorithmic enhancement modules. Given an equal weight of each slice of data in the input AOD tensor, the reconstructed data fields from the benchmark method were prone to resembling a mean state determined largely by the principal mode of the input tensor. In this context, peak values in the target image might be underestimated (or overestimated for low values) because of relatively few soft data resembling similar patterns in the input tensor (e.g., Fig. 2c).

By incorporating the attention mechanism, each slice of data in the raw AOD data cube was adaptively weighted, with greater weights given to those with broader spatial coverage and closer similarities to the target AOD_Terra image. This strategy is vital for reducing contributions from irrelevant data, particularly when encountering imbalanced data samples within the raw AOD data cube, i.e., more irrelevant data and fewer similar images. Moreover, the importance of the target image was maximized during the tensor completion procedure by assigning a 100 % weight. Compared to the benchmark method, extreme values in raw AOD_Terra images were better reconstructed using the method embedding the attention mechanism. For instance, in Fig. 2b, the benchmark method apparently overestimated low AOD values in the north, whereas such a discrepancy was largely mitigated using methods involving the attention mechanism.

Figure 4 Data accuracy of daily gap-free AOD grids in the LGHAP v2 dataset compared to AOD observations from AERONET across the globe during 2000–2021. Note that the AERONET AOD observations were independent data and had not been used in the gap-filling process.

In contrast to the benchmark method, which used an invariant background throughout the tensor completion process, an adaptive background updating scheme was incorporated here to accelerate the convergence speed and mitigate possible error propagation arising from numerical simulations to the final reconstruction fields. Compared to the benchmark method, as illustrated in Fig. S5, the adaptive background updating module enabled us to reduce the adverse impact of manually added outliers in raw background fields, thereby avoiding large error propagation from background fields into the reconstructed AOD data. Although the better quality of the reconstructed fields derived from the improved methods demonstrates the efficacy of these two newly developed algorithmic enhancement modules, the benefits could be largely cancelled out when confronted with images containing excessive data gaps (e.g., Fig. 2c). The inherent reason could be attributed to few observational data in the target image for reference to leverage the attention mechanism to pinpoint similar AOD images from the historical data series.

In Fig. 3, we evaluated the impact of the missing rate of the target image on the AOD gap-filling accuracy. By masking one truly observed AOD_Terra image with arbitrarily selected cloud masks, a series of target images at different missing rates, as shown in the top row of Fig. 3, were simulated for gap-filling trails. As shown, the reconstructed fields agreed fairly well with the observed AOD fields, resembling the actual AOD distribution over the outlined region well, even in extreme situations with excessive data gaps, demonstrating an excellent performance of the proposed gap-filling method. As expected, the accuracy of the reconstruction fields decreased along with an increase in the missing rate. For instance, when the missing rate was greater than 80 %, the low values in the upper left of the raw AOD_Terra image were not properly reconstructed, largely because of the limited prior knowledge in the target image for use when constructing the raw AOD tensor. This effect also highlights the crucial importance of prior information for the gap-filling accuracy. Therefore, increasing prior information is the most promising way to improve the gap-filling accuracy, in particular for regions with substantial data gaps.

Table 2 An intercomparison of AOD data accuracy between satellite-based retrievals (raw MAIAC AOD), numerical aerosol diagnostics (downscaled MERRA-2 AOD), purely reconstructed data, and the final gap-free product (LGHAP v2 AOD) by comparing AOD observations from AERONET across the globe during 2000–2021. Note that the term “purely reconstructed AOD” refers to the imputed AOD estimates, while “LGHAP v2” refers to the gap-filled AOD dataset combining both satellite-based retrievals and purely reconstructed data. The expected error (EE) envelope for AOD over land was defined as ±(1.5 × AOD_AERONET+0.05).

4.2 Data accuracy of global gap-free AOD in LGHAP v2

The gap-free AOD grids in LGHAP v2 were generated by filling in data gaps in AOD_Terra images with reconstructed AOD estimates at each collocated footprint over land. In comparison to the independent AOD observations from AERONET, the data accuracy of the gap-free AOD in LGHAP v2 was comprehensively evaluated across the globe. Figure 4a–c present the spatial distribution of the site-specific correlation coefficient (R), root mean square error (RMSE), and bias between AOD in the LGHAP v2 and AERONET observations, respectively. Regardless of the uneven distribution of ground-based aerosol observing stations and variations in data samples between sites, the ground validation results indicate good agreement between the AOD in the LGHAP v2 and AERONET observations, with a site-specific R of 0.76 ± 0.14 and an RMSE of 0.09 ± 0.08 on a global scale. Note that site-specific data accuracy metrics vary across regions, with larger biases mainly observed in central and East Asia as well as in Africa – regions always suffering from high aerosol loadings.

Figure 4d–i present scatter plots between the LGHAP v2 AOD and AERONET observations in six major continental regions. As shown, the reconstructed AOD estimates were prone to an underestimation of large AOD values (> 0.80) versus an overestimation of low values (< 0.2) across these six regions. This effect is particularly common in machine learning, largely because of the imbalanced distribution of data values in the training samples (Johnson and Khoshgoftaar, 2019; Shi et al., 2022). A similar reason could also be applied for the tensor completion as the missed AOD extremes may not be accurately reconstructed to their nominal levels; instead, they tend to resemble a mean state that was determined by principal modes via a low-rank approximation.

Figure 5 Temporal variations in the monthly AOD over six AERONET sites with long-term AOD observations from 2000 to 2021. The panels on the right present histograms of AOD deviations between the LGHAP v2 and AERONET observations at each individual site.

To further verify the data accuracy of the imputed AOD estimates, we compared the gap-filled AODs in the LGHAP v2 dataset with two major gridded products of AOD_Terra and AOD_M2. As shown in Table 2, the purely reconstructed AOD estimates have an R of 0.83 and an RMSE of 0.15 compared to the AERONET AOD observations at the global scale – comparable to the data accuracy of AOD_M2 (R = 0.83, RMSE = 0.14) but lower than that of AOD_Terra (R = 0.88, RMSE = 0.11). Nevertheless, the imputed AOD estimates achieved comparable data accuracies to AOD_Terra in Africa (R = 0.80, RMSE = 0.20) and Australia (R = 0.62, RMSE = 0.08), largely because of the availability of abundant satellite-based AOD prior information (refer to the AOD coverage ratio shown in Fig. S1 in the Supplement) to facilitate AOD tensor completion. In contrast, the LGHAP v2 AOD estimates in Europe and Asia have poorer data accuracies relative to AOD_Terra, particularly in East Asia. The possible reasons could be extensive missing values, severe aerosol pollution levels, and significant spatial variations in aerosol loadings over these regions. Compared to AOD_Terra, the gap-filled AOD data tended to overestimate the AERONET AODs (17.59 % versus 11.45 % above the envelope of expected error), resulting in an even larger global mean AOD (0.19 versus 0.17) and implying that a greater number of large AOD values were reconstructed in the imputed AOD estimates. Moreover, the accuracy of LGHAP v2 AOD data outperforms that of the gap-filled AOD dataset (R²=0.6031 and RMSE = 0.1350) generated by Guo et al. (2023), in which missing AODs in AOD_Terra were predicted using various proxy variables (e.g., meteorological factors and population density) via a random forest model.

In Fig. 5, we compared temporal variations in AOD between the LGHAP v2 dataset and ground-based observations at six AERONET sites with long-term records. Compared to discrete AOD observations from AERONET, the gap-free AOD time series accurately reconstructed long-term variations of aerosol loading from 2000 to 2021 at these monitoring sites, with R ranging from 0.83 to 0.97 and RMSEs varying between 0.04 and 0.24. Note that the large RMSEs observed at the Alta Floresta and Beijing sites are more likely ascribed to the reconstruction failures of abnormal AOD peaks, largely because of very limited peak values for reference in the AOD tensor. Referring to histograms of AOD deviations between the LGHAP v2 and AERONET observations, more than 80 % of the AOD biases fell within the range of −0.1 to 0.1, demonstrating a high accuracy of gap-filled AOD in the LGHAP v2 dataset.

Figure 6 Site-based validation accuracy of PM_2.5 concentration estimates derived from gap-free AOD images using the proposed SCAGAT method. (a) Scatter plots between PM_2.5 estimates derived from the SCAGAT model and the withheld PM_2.5 concentration measurements. (b) Same as panel (a) but for gap-free PM_2.5 estimates fusing ground-measured PM_2.5 concentration measurements. (c–d) Site-based correlation coefficient and RMSE, respectively, for LGHAP v2 PM_2.5 concentrations. (e–g) Histograms of the LGHAP v2 PM_2.5 concentration bias over China, the United States, and Europe, respectively. Note that the ground-based PM_2.5 concentration data used here for validation were not used in the model training or in the data fusion procedures.

4.3 Data accuracy of global gap-free PM_2.5 concentrations in LGHAP v2

Global gap-free PM_2.5 concentration estimates were derived from gap-filled AOD images by taking advantage of the novel SCAGAT method that was specifically developed for global PM_2.5 concentration mapping. Additional details of the SCAGAT method were provided in another study (Li et al., 2024), and here we focused on the data accuracy of the global gap-free PM_2.5 concentration estimates. Figure 6 presents the validation accuracy of the daily gap-free PM_2.5 concentration estimates by comparing them to the ground-based PM_2.5 concentration records measured at 350 former holdout sites. As indicated, by accounting for the spatial representativeness of the prediction models during the spatial extrapolation, PM_2.5 concentration estimates derived from the SCAGAT model are in better agreement with ground-based PM_2.5 concentration measurements, with an R of 0.91 and an RMSE of 9.587 µg m⁻³, surpassing the performance of our traditional machine-learned models (Bai et al., 2019, 2022a, 2023). Meanwhile, the data accuracy was further improved by correcting modeling biases using sparsely distributed in situ PM_2.5 concentration measurements via optimal interpolation, resulting in an improvement in R to 0.95 and a decrease in RMSE to 5.7 µg m⁻³ (Fig. 6b). As shown in Fig. 6e, the PM_2.5 concentration estimates over China in LGHAP v2 have a higher data accuracy (R = 0.97, RMSE = 7.93 µg m⁻³) than those in LGHAP v1 (R = 0.95, RMSE = 12.03 µg m⁻³). Figure 6c–d present a site-based distribution of R and RMSE for the LGHAP v2 PM_2.5 concentrations over each individual validation site. Compared to the United States of America and Europe, as depicted in Fig. 6e–g, larger PM_2.5 concentration biases were observed in China because of higher PM_2.5 loadings therein.

Table 3 presents the data accuracy of the gap-free PM_2.5 concentrations in the LGHAP v2 dataset during the period of 2000–2021 over nations with sufficient records of ground-based PM_2.5 concentration measurements. It indicates that the data accuracy of PM_2.5 concentration estimates varied across regions, with R changing from 0.71 to 0.98 and RMSEs ranging between 1.15 and 32.69 µg m⁻³. Regardless of the substantial differences in the total number of data pairs, larger RMSEs are mainly observed in regions like Mongolia (32.69 µg m⁻³) and India (25.34 µg m⁻³), which often suffered from severe PM_2.5 pollution episodes. The spatially varying accuracy metrics highlight the great complexity in large-scale PM_2.5 modeling, which also underscores the critical importance of accounting for spatial representativeness when applying models over other regions for data extrapolation.

In Fig. 7, we examined long-term variations in PM_2.5 concentrations in four different cities from 2000 to 2021. Good agreement with the previously withheld PM_2.5 concentration measurements demonstrated a high accuracy of the LGHAP v2 PM_2.5 concentration estimates. Compared to temporally discrete PM_2.5 concentration records measured by ground monitors, the gap-free LGHAP v2 PM_2.5 concentration time series enabled us to better understand the long-term variability of haze pollution across the globe. As shown, declining trends were observed in PM_2.5 concentrations as early as 2006 in New York (United States), whereas apparent reductions were mainly observed after 2012 in Jilin (China) and 2015 in Toyama (Japan). Overall, the gap-free and high-accuracy merits render PM_2.5 concentrations in the LGHAP v2 dataset reliable data sources for assessing long-term trends of haze pollution across the globe.

Table 3 The data accuracy of gap-free PM_2.5 concentrations in the LGHAP v2 dataset compared to ground-based measurements in countries with sufficient PM_2.5 records. N denotes the total number of PM_2.5 concentration data pairs for calculating R, RMSE, and bias.

Figure 7 An intercomparison of temporal variations in monthly mean PM_2.5 concentrations in four different cities between the LGHAP v2 and collocated ground-based PM_2.5 concentration measurements from 2000 to 2021.

Figure 8 Spatial distribution of the global annual mean PM_2.5 concentrations derived from the LGHAP v2 dataset between 2000 and 2021.

Figure 8 presents the temporal variations in the global annual mean PM_2.5 concentration distribution from 2000 to 2021. As shown, the daily gap-free LGHAP v2 dataset seamlessly supports the derivation of comparable annual mean PM_2.5 concentration maps between years, and data-gap-related biases in raw AOD_Terra images were eliminated. Meanwhile, the quality-assured annual mean PM_2.5 concentration maps enable us to easily pinpoint the hotspot regions suffering from severe haze pollution and to analyze the long-term variability of global PM_2.5 concentrations. Specifically, Mongolia, northern India, eastern China, and central Africa were identified as four major regions with relatively high PM_2.5 loadings, in particular northern India, becoming a hotspot region suffering from more severe PM_2.5 pollution on the planet. Substantial PM_2.5 reductions were observed in eastern China from 2014 onwards, with PM_2.5 concentrations reduced to levels even comparable to countries in central Asia.

Figure 9 An illustration of AOD responses to wildfires, sandstorms, and haze pollution episodes across the globe, as characterized by gap-free AOD in the LGHAP v2 dataset. The global map in the middle panel shows the spatial distribution of major land cover types in 2020.

5 Discussion

Spatially contiguous AOD and PM_2.5 concentration grids are pivotal in regional air quality management, haze pollution exposure risk assessment, and aerosol radiative forcing diagnosis. By seamlessly gearing up state-of-the-art machine learning and tensor completion methods, a novel big Earth data analytics framework was developed to fulfill the generation of long-term high-resolution AOD and PM_2.5 concentration grids (LGHAP v1) in our previous study (Bai et al., 2022a). Specifically, multimodal AODs and relevant air quality data acquired from diverse satellites, numerical models, and ground monitoring stations were first harmonized using random forest models. Next, multisource AOD data flows were weaved neatly as the tensor inputs, with data gaps in daily MODIS AOD images properly reconstructed via low-rank tensor completion. Finally, gap-free PM_2.5 concentration grids were mapped from gap-filled AOD images using a random forest model. This big data analytics framework provided an effective solution to integrate multimodal Earth observations from diverse sources to generate high-quality AOD and PM concentrations in China.

In this study, aiming to generate global gap-free AOD and PM_2.5 concentration grids, i.e., the LGHAP v2 dataset, the previous big Earth data analytics framework was adopted but enhanced with several new features, with particular focuses on accommodating the rocketing data size and global-scale modeling demand other than reducing modeling biases. Specifically, an attention mechanism, inspired by deep-learning techniques, was hereby introduced to weight each data slice in the input tensor to account for the drawback induced by the equal weight strategy, with larger weights assigned to data slices with fewer data gaps and more similar to the target image. In other words, both the spatial coverage ratio of valid observations in all soft data and the mutual information between the target and soft data were considered simultaneously to weight each data slice in the AOD tensor. A weighted AOD tensor was then calculated for tensor completion instead of using all the available information in the AOD tensor indifferently. Although the ablation experiments shown in Fig. 2 have demonstrated the efficacy of this attention-reinforced tensor construction strategy, the underlying philosophy, in particular the relative importance of mutual information and extra spatial coverage, has not yet been fully justified and assessed.

An adaptive background field updating scheme was also introduced to iteratively update prior information in the target AOD images. Compared to the invariant prior information, adaptively updated prior information allowed for mitigation of the influence of uncertainties in the prior information on the reconstruction accuracy, particularly large modeling biases from numerical simulations. Despite these algorithmic improvements, a slightly reduced data accuracy of gap-filled AODs in China from the LGAHP v2 dataset was observed compared to those in the LGHAP v1 dataset. Further investigations revealed that this was mainly due to the relatively poor data accuracy of the downscaled AOD_M2 data because a global-scale versus regional downscaling model was applied. Nonetheless, benefiting from the adaptive background updating scheme, the modeling biases in AOD_M2 were effectively suppressed in the final reconstructed AOD fields, evidenced by larger biases of AOD_M2 (R = 0.77, RMSE = 0.36) versus smaller biases of the purely reconstructed AOD (R = 0.82, RMSE = 0.26).

The global gap-free and high-resolution benefits render the LGHAP v2 dataset a promising data source to monitor global aerosol distribution and variations in space and time. As illustrated in Fig. 9, aerosol-related environmental disturbance episodes, e.g., sandstorms, wildfires, or haze pollution events, can be indicated well by local rising AODs. More importantly, the gap-filled AOD dataset provides us with an unprecedented opportunity to monitor aerosol loadings and variations even under cloud cover, e.g., the haze pollution episodes over southern India and eastern China shown in Fig. 9d and e. This largely benefited from the intelligent spatiotemporal pattern recognition as well as the assimilation of air quality measurements from ground monitoring stations and numerical aerosol diagnostics. While this global air quality mapping approach greatly facilitates the surveillance and management of air pollution around the world, the LGHAP v2 dataset would also significantly reduce uncertainties in the health-related aerosol exposure risk assessment results because of the gap-free and high-resolution advantages.

Figure 10 AOD trends over 12 regions of interest worldwide from 2000 to 2021 estimated from gap-free AODs in the LGHAP v2 dataset. The top row shows the spatial distribution of global AOD deviations between the first and second decades in the 21st century. The 12 diagrams in the bottom panel show the linear trend of mean AOD over the outlined region of interest at different starting times with varying time window sizes.

Global AOD variation trends were carefully examined by taking advantage of the LGHAP v2 AOD dataset. Figure 10a presents the AOD deviations between the AOD averages during the first and second decades of the 21st century across the globe. As shown, substantial AOD increases in the 21st century are primarily present over India and central Africa, with remarkable AOD decreases observed in the middle of South America. In North America, AOD increases were mainly observed in Canada and the western United States, whereas AOD decreases were found in the eastern United States. Additionally, in reference to temporally varying AOD trends in regions A and B, evident AOD increasing trends were observed in the United States from 2012 onwards, while significant decreasing trends in the eastern United States were entirely reversed after 2015. This effect could be partially attributed to more frequent and intensive wildfire emissions in North America during the second decade of the 2000s (Burke et al., 2023; X. Wei et al., 2021). A similar effect was also observed in Europe, with an apparent slowdown in the AOD decreasing trend after 2010.

Inverse effects were also observed in China but with totally different temporal transition patterns. As shown, statistically significant AOD increasing trends were observed in eastern and southern China in the first decade, with a slowdown starting around 2007, followed by a sudden reversion to decreasing trends after 2010. This was also the most significant AOD decreasing trend during the 2010s around the world. This observational evidence confirms the success of clean air action in improving air quality in China during recent decades (Bai et al., 2022a; Liang et al., 2020; Zhang et al., 2019). A similar temporal variation pattern was also observed in the Middle East but with relatively weak trends. In contrast, India was a hotspot area showing an increasing trend in AOD throughout the 2000s, despite a short period of increasing hiatus from 2013 to 2015.

Global gap-free PM_2.5 concentrations were derived based on gap-filled AOD grids by taking advantage of a novel SCAGAT model. Unlike many other data-driven models, the spatial representativeness was accounted for in the SCAGAT model, providing a unique solution to modeling PM_2.5 concentrations over regions even without PM_2.5 monitoring sites. Daily gap-free PM_2.5 concentration grids favor the assessment of the pandemic's influence on regional air quality. Figure 11a and b present the spatial distribution of PM_2.5 concentrations before and during the COVID-19 pandemic, respectively. Neglecting long-term variation trends in PM_2.5 concentrations, the substantial PM_2.5 decreases in central and eastern China as well as in central Europe clearly indicate the positive effect of pandemic-related mobility restrictions on air quality improvement (by comparing PM_2.5 concentrations in 2019 and 2020 during the synchronous period). In contrast, PM_2.5 reductions were relatively small in the United States due to the lack of mobility restriction measures, with apparent PM_2.5 reductions observed mainly in regions like Chicago. Overall, the LGHAP v2 dataset enables us to better investigate global aerosol variations and assess PM_2.5-related health exposure risks.

Figure 11 Influence of the COVID-19 pandemic on PM_2.5 concentrations in the United States, Europe, and China. PM_2.5 concentrations from LGHAP v2 were averaged over synchronous periods in 2019 and 2020 for intercomparison.

6 Data availability

The LGHAP v2 dataset provides global gap-free AOD and PM_2.5 concentration grids from 2000 to 2021 with a daily 1 km resolution. To facilitate the data sharing, each daily map was saved as a single NetCDF file, and the data in each individual month were then archived as one zip file. Table 4 summarizes the permanent digital object identifiers for data in each calendar year from 2000 to 2021. All these datasets are publicly available at the LGHAP community link via https://zenodo.org/communities/ecnu_lghap (Bai and Li, 2023a). The data user guide and visualization codes (Python, MATLAB, R, and IDL) were also provided to guide the users in retrieving data from the NetCDF files, which can be accessed at https://doi.org/10.5281/zenodo.10216396 (Bai and Li, 2023b).

Table 4 List of data links for AOD and PM_2.5 concentration grids in the LGHAP v2 dataset for each individual year.

7 Conclusions

In this study, the LGHAP v2 dataset, a successor of LGHAP v1, was generated to provide global gap-free AOD and PM_2.5 concentration grids with a daily 1 km resolution from 2000 to 2021, by leveraging an improved big Earth data analytics approach. The ground validation results confirm high accuracies of these two gap-free products, with AOD having an R of 0.85 and an RMSE of 0.14 compared to the AERONET AOD observations, which are slightly worse than the original MCD19A2 product (R = 0.88 and RMSE = 0.11). Similarly, PM_2.5 concentration estimates derived from gap-free AOD via the SCAGAT method show agreement with the withheld ground-based PM_2.5 measurements, achieving an R of 0.91 and an RMSE of 9.57 µg m⁻³, while the data accuracy was improved to an R of 0.95 and an RMSE of 5.7 µg m⁻³ with the fusion of ground-measured PM_2.5 concentrations.

Several new algorithmic enhancement modules were incorporated into the big data analytics framework to improve both the computing speed and the reconstruction accuracy. The ablation experiments demonstrated the effectiveness and advantages of the newly implemented attention mechanism in weighing each slice of soft data in the AOD tensor. Updating prior information in the target image after each tensor reconstruction iteration helped mitigate the risk of error propagation from numerical aerosol diagnostics to the final reconstructed field and improve the convergence speed of tensor completion. Overall, this study provides a compelling illustration of big Earth data analytics to generate high-quality remote sensing datasets by synergistically integrating and assimilating multimodal data from diverse sources via machine learning techniques. Additionally, this big data analytics approach could also be used for near-term gap-free AOD mapping by simply replacing numerical AOD reanalysis with forecasting fields (e.g., CAMS forecasts).

This study also provides new insights into how to deal with the scale problem when developing large-scale environmental variable (e.g., PM_2.5 concentration) mapping models. Instead of constructing a global model with all paired data samples, site-specific PM_2.5 prediction models were first established using a random forest model, and a graph attention network was then developed to establish an ensemble learning model to integrate multiple PM_2.5 estimates derived from site-specific random forest models trained over sites with similar scene features to the target grid. By accounting for the scene similarity between geographic regions, the proposed deep-learning model attempted to address the scale problem in large-scale PM_2.5 modeling practices.

The LGHAP v2 dataset is publicly accessible using the aforementioned links. The gap-free and high-resolution dataset can be used as a reliable data source for assessing aerosol–climate interactions as well as PM_2.5 exposure risks and related health outcomes around the world. Researchers are also encouraged to use this dataset to evaluate the status and trends of urban aerosol pollution across the globe to support the assessment of Sustainable Development Goals.