After carbon dioxide (CO2), methane (CH4) and nitrous oxide (N2O) are currently the two most important anthropogenically emitted greenhouse gases (GHGs), and their concentrations in the atmosphere are increasing at substantial rates e.g.. The primary natural sources of CH4 are wetlands, the decay of organic waste and livestock, whereas anthropogenic sources include landfills and the production and transport of coal, natural gas and oil e.g.. The primary emission sources for N2O are agricultural practices, industrial activities, the combustion of fossil fuels and the treatment of solid/liquid waste e.g.. Importantly, as measured by the global warming potential (GWP), CH4 is about 25 times and N2O is about 300 times more potent as GHGs compared to CO2.

The lifetime of CH4 in the troposphere is about 9 years e.g., and it is primarily removed through oxidation by OH. However, in the stratosphere, CH4 destruction is much slower; hence, its local lifetime increases to about 150 years . CH4 oxidation is also an important source of water vapour in the stratosphere, which plays a key role in ozone chemistry via HOx cycles, so it also influences the radiative balance in the middle stratosphere. The primary atmospheric sink for N2O is photolysis (producing N2 + O) in the stratosphere/mesosphere, so it is also a long-lived species (lifetime about 120 years ). A secondary sink for N2O is the reaction with O(1D) to produce NO, which plays an important role in the middle-atmosphere O3 budget via the NOx cycle. An important aspect is that increases in both OH and NO can also have positive impacts on ozone, especially in the lower stratosphere, as they help to convert reactive species to long-lived reservoir species. For example, OH + NO2 (+ M) leads to HNO3 formation, while CH4 + Cl leads to HCl formation, reducing concentrations of reactive NO2 and Cl. Additionally, as both CH4 and N2O are long-lived in the stratosphere, monitoring their concentrations also helps us to understand changes in stratospheric chemistry and dynamics.

However, despite their importance, there are only a few satellite instruments that provide global stratospheric profiles of CH4 or N2O. Relatively long-term and high-quality data records are available from two solar occultation instruments, the HALogen Occultation Experiment (HALOE) and the Atmospheric Chemistry Experiment – Fourier Transform Spectrometer (ACE-FTS), and from limb sounding instruments such as the Michelson Interferometer for Passive Atmospheric Sounding (MIPAS) and the Microwave Limb Sounder (MLS). However, these instruments have different spatial and temporal coverages and they use different measurement techniques and retrieval algorithms. Hence, merging these satellite data to construct a single long-term data set for a given species is quite challenging.

Therefore, although stratospheric CH4 and N2O profile data sets were released recently by , they did not attempt to merge data from different satellite instruments. Briefly, these data sets were released as part of the Stratospheric and Tropospheric Processes And their Role in Climate (SPARC) Data Initiative and contain monthly mean zonal mean profiles in volume mixing ratio (vmr) units at pressure levels. Data from individual satellite instruments are averaged at 36 latitude bins (2.5∘ latitudinal resolution) and provided at 26 pressure levels ranging from 300 to 0.1 hPa. SPARC CH4 profile data are constructed using ACE-FTS (2004–2019), HALOE (1991–2005) and MIPAS (2002–2012) measurements . Note that SPARC data use an earlier (v3.6) version of the ACE-FTS data. For N2O, there is no data set for the 1990s, but for later periods, SPARC N2O data contain monthly mean values from Aura-MLS (based on v4.2), MIPAS (v224), the Sub‐Millimetre Radiometer (SMR, v2.1) and ACE-FTS measurements. Monthly means values are available only if there are more than five valid profiles for a given latitude/altitude range. Monthly mean files are available for individual instruments and there is no merging or adjustment for different data sets.

To our knowledge, until now, no attempt has been made to merge satellite data records to construct long-term stratospheric CH4 and N2O profile data sets. Here, we do this by constructing correction terms for the stratospheric CH4 and N2O profiles from a chemical transport model by analysing the difference between the model and available satellite observations. Then, the correction terms (i.e. the differences that are needed to adjust the TOMCAT CH4/N2O profiles) are calculated for all the model grid points to construct a long-term, gap-free stratospheric profile data set. Details of the satellite data and model setup used here are given in Sects. 2 and 3, respectively. The methodology used to estimate the correction terms is described in Sect. 4. An evaluation of the newly constructed data sets for CH4 and N2O is presented in Sect. 5. Details of data availability are given in Sect. 6, followed by a summary and the conclusions in Sect. 7.

Given that CH4 and N2O are potent GHGs and primary sources of stratospheric water vapour and NOx, stratospheric measurements of CH4 and N2O gained scientific attention even before the discovery of the Antarctic ozone hole . Initial measurements were performed by the Stratospheric and Mesospheric Sounder (SAMS) instruments on the Nimbus 7 satellite that was launched in 1978 . Similarly, the Atmospheric Trace Molecule Spectroscopy (ATMOS) instrument provided about 350 profiles during four space shuttle missions (in 1985, 1992, 1993 and 1994). Later, the Improved Stratospheric and Mesospheric Sounder (ISAMS) was able to provide about 2600 profiles per day for about 180 d between 1991–1992, but retrieval was feasible only for the upper stratospheric/mesospheric altitude range e.g..

A step-change in the number of stratospheric CH4 measurements occurred with the deployment of HALOE on the Upper Atmosphere Research Satellite (UARS) in September 1991, followed by ACE-FTS in August 2003. Both instruments provided about 30 profiles per day (discussed below). Two additional instruments, SCIAMACHY (SCanning Imaging Absorption spectroMeter for Atmospheric CHartographY) and MIPAS on the Envisat satellite platform, also provided useful stratospheric CH4 profiles over the 2003–2012 time period e.g.. For N2O, the Cryogenic Limb Array Etalon Spectrometer (CLAES) on the UARS satellite platform provided about 1 year of profile measurements (October 1991 to July 1992). Later, the Sub-Millimetre Radiometer (SMR) on Odin, launched in 2001 e.g., MIPAS, and MLS on the Aura satellite also provided very useful N2O profile measurements. However, to avoid inter-instrument biases, which are likely due to differences in the measurement techniques, we decided here to use only HALOE and ACE-FTS data.

As both CH4 and N2O are long-lived tracers in the stratosphere, their distributions in this region are largely determined by the transport process. Hence, we decided to use profiles simulated by the TOMCAT CTM as it is forced with an up-to-date meteorological reanalysis data set. Briefly, TOMCAT is an off-line three-dimensional CTM that includes a comprehensive stratospheric chemistry scheme, but, in the version used here, with a simple tropospheric chemical scheme . This means that concentrations of long-lived ozone-depleting substances (ODSs) and GHGs are prescribed as surface mixing ratio boundary conditions e.g., and these species are assumed to be well mixed throughout the troposphere. For CH4, the model uses observed monthly mean global surface concentrations from the National Oceanic and Atmospheric Administration (NOAA) network. The CTM setup is therefore similar to the control simulations used in our recent studies such as and . The model simulation is performed at a 2.8∘  × 2.8∘ horizontal resolution with 32 hybrid sigma-pressure levels (surface to about 60 km) and is forced with ERA5 (and ERA5.1) reanalysis meteorology . The effects of time-varying solar flux changes and volcanically enhanced stratospheric aerosol are included by using separate time-varying forcing files e.g..

For stratospheric ozone, various methodologies have been adopted to merge different types of data to construct homogenized data sets. They include both simple and complex methodologies for adjusting biases for overlapping time periods e.g. and the use of multivariate linear models e.g. and data assimilation . However, we are not aware of any attempt to construct long-term stratospheric CH4 and N2O profile data sets using different satellite data sets.

Here, our approach is similar to that of for ozone, who used CTM profiles as a transfer function and estimated model-observation biases using machine learning. However, they used observation-based monthly mean zonal mean ozone values from the Stratospheric Water and OzOne Satellite Homogenized (SWOOSH) data set rather than individual satellite data products. As there are a number of satellite instruments that provide ozone profile measurements, monthly mean zonal mean values in merged ozone data sets are considered to be well constrained. However, as noted in Sect. 2, there are very few satellite instruments that provide CH4 profile measurements (mainly two occultation instruments providing 30 profiles per day), so in this study we decided to use individual data points to train a machine learning algorithm. Similarly, for N2O (among occultation instruments), only ACE-FTS provides a long profile data record, but again it has limited spatial coverage; hence, the calculation of monthly mean zonal mean profiles is subject to sampling errors.

Overall, there are six steps in our approach. First, TOMCAT output fields are sampled for HALOE and ACE measurement collocations. There are about 95 000 HALOE profiles and over 106 000 ACE profiles in the 1991–2021 time period. Second, as ACE profiles are available at 1 km vertical resolution, HALOE profiles are also binned at 1 km vertical resolution and TOMCAT profiles (surface to 60 km) are interpolated to the same grid.

Third, we calculate observation–TOMCAT profile differences for each 1 km grid and satellite measurements are included only if retrieval errors are less than 100 % and retrieved values are greater than zero. Note that we assume that all the measurements with retrieval errors less than 100 % are more or less the absolute truth. Hence, no other uncertainties are considered in the further calculations. Our attempt is to construct profile data that would approximate HALOE/ACE data if the instruments had denser measurements without any temporal gaps. As there are distinct dynamical (and chemical) regimes in the stratosphere in terms of processes controlling the distribution of these two GHGs, we divide global measurements into five latitude bins: Southern Hemisphere (SH) polar (SHpol, 50–90∘ S), SH mid-latitude (SHmid, 20–70∘ S), tropical (40∘ S–40∘ N), Northern Hemisphere (NH) mid-latitude (NHmid, 20–70∘ N) and NH polar (NHpol, 50–90∘ N). A 20∘ (10∘ from either side) latitudinal overlap between the bins is allowed to include possible extreme variations in the training data set. Estimated differences for overlapping grids are averaged in order to avoid possible sharp edges near the latitude bin boundaries.

Fourth, we train the extreme gradient boosting (XGBoost) regression model for TOMCAT–observation differences of CH4 or N2O for each vertical level. This means there is a separate model for each vertical level (46 for 15–60 km) for each of the five latitudinal bins. Briefly, XGBoost is a supervised machine learning algorithm that uses an ensemble of decision trees e.g.. XGBoost applies the principle of boosting weak learners using the gradient descent architecture (gradient boosting) with some additional components such as L1 and L2 (Lasso and Ridge) regularization, which helps to prevent over-fitting. There are 13 explanatory variables (or features) in our XGBoost regression model taken from TOMCAT output fields or the ERA5 reanalyses used to force the model. For example, the XGBoost regression model for CH4 can be represented as 1dCH4=β1CH4+β2O3+β3N2O+β4HNO3+β5HCl+β6H2O+β7HF+β8NO2+β9ClONO2+β10T+β11PV+β12Θ+β13t+ϵ, where T and PV are the temperature and potential vorticity from ERA5 at co-located TOMCAT grid points. Measurement latitude (Θ) and date (t) variables are included to represent temporal/spatial variations in the measurements, whereas ϵ denotes unexplained errors. Each variable β1 to β13 can be considered the contribution coefficient for a given explanatory variable. For CH4, we include an additional (14th) step-function-like term in the XGBoost model that is set to 0 for the HALOE time period and 1 for the ACE-FTS time period. Our approach here is to assume that nearly all differences in the TOMCAT CH4 or N2O profiles with respect to HALOE and ACE data arise from the incorrect representation of the chemical and dynamical processes in the CTM (including inhomogeneities in ERA5 data that are used to drive TOMCAT transport). Our aim is to find correction terms for the TOMCAT CH4 or N2O profiles so that they match observational profiles for a particular distribution of model tracers and dynamical setup. Hence, we include nine tracers of varied lifetimes (i.e. CH4, O3, N2O, HNO3, HCl, H2O, HF, N2O, ClONO2) from TOMCAT. We are aware that some tracers are correlated as all the variables are from a TOMCAT simulation (or forcing meteorology); hence, we use the Lasso (L1) regularization option to remove less important variables in case one or some of them are highly correlated at a particular level. We use the Python package XGBoost (https://xgboost.readthedocs.io/en/stable/python/python_intro.html, last access: 5 January 2023) for the analysis with the following options: n_estimators = 1000, max_depth = 4, alpha = 0.3, learning_rate = 0.1, min_child_weight = 6. As mentioned earlier, profiles prior to 2018 are used to train (70 %) and test (30 %) XGBoost for individual vertical levels. As an additional check, we use the last 3 years (2019–2021) of data points for the evaluation.

Fifth, we sample the daily TOMCAT output at 01:30 and 13:30 UTC equatorial crossing times (daytime and nighttime sampling). The TOMCAT 3D fields are then re-gridded at 1 km vertical resolution before dividing them into five latitude bins (see above). Trained XGBoost regression models are then used to calculate correction terms for all twice-daily 3D output profiles.

Sixth, correction terms for individual model grid points are merged to construct twice-daily (01:30 and 13:30 UTC) 3D (longitude/latitude/height) correction terms. As mentioned above, we use simple averaging for the overlapping grid points to avoid sharp boundaries, followed by simple three-dimensional (latitude–longitude) smoothing using three-point boxcar smoothing. These twice-daily correction terms are then added to the original TOMCAT CH4 and N2O profiles. Daily mean 3D (longitude/latitude/height) correction terms are calculated by averaging the 01:30 and 13:30 UTC fields, and then zonal means (latitude/height) are calculated to produce daily mean zonal mean TCOM-CH4 and TCOM-N2O profiles.

As noted in the “Introduction”, CH4 and N2O concentrations in the lower stratosphere are largely controlled by dynamical processes. The reanalysis data sets used to drive transport in the CTM can be considered as our best knowledge of the past atmosphere as they attempt to incorporate most of the available high-quality meteorological observations using data assimilation. However, they are prone to issues related to changes in the number and type of observations assimilated in the reanalysis system, which might introduce inhomogeneities into the data sets produced. On the other hand, although chemical models are ideal tools for simulating and understanding past changes in these two greenhouse gases using consistent chemical schemes, they are also prone to deficiencies. For example, some computationally expensive processes (e.g. vertical mixing in the troposphere) are represented by somewhat simplified parameterizations. Additionally, most of the chemical reaction rates (loss rates) calculated in the model scheme can also have large uncertainties. Hence, chemical-transport-model-simulated profiles often show some kind of bias with respect to observational data sets. Similarly, although occultation-technique-based instruments measure atmospheric spectra at relatively high resolution, they also include simplified parameterizations for complex radiative processes (e.g. scattering, the contribution from interfering gases), and so retrieval errors are also sensitive to changes in stratospheric conditions. Hence, here we assume that some of the differences between TOMCAT and observations can be attributed to the distribution of other TOMCAT tracers. We use XGBoost to identify possible interconnection patterns between TOMCAT CH4 or N2O differences and other tracers using available data points so that corrections can be estimated for all model grid points.

Figure 1 shows vertical profiles of estimated variance (R2) and feature (explanatory variable) importances for the SHpol (50–90∘ S) latitude bin for the XGBoost regression model. Feature importance can be considered as a regression coefficient indicating how much a given variable contributes towards the CH4 or N2O bias-correction prediction. Variance and feature importances for SHmid, tropics, NHmid and NHpol are shown in Figs. S1 to S4 in the Supplement, respectively. For SHpol, XGBoost seems to show excellent performance for both species throughout the stratosphere, with R2 values ranging from 0.6 to 0.8. This also validates our approach of using different long-lived tracers as variables in the regression model. As expected, concentrations of long-lived tracers seem to show close relationships to the biases seen in CH4 and N2O profiles. However, Figs. S1 to S4 show that R2 values for other latitude bins are somewhat smaller (near 0.5), indicating that regions with less dynamical variability (e.g. mid-latitudes) might need some additional features that are not included in this setup.

Another important aspect is that R2 values for CH4 remain almost flat between 25 to 50 km, but for N2O, R2 values are close to 0.6 in the lower stratosphere and lower mesosphere, with minima near 30 km. The time (date) term is included in the XGBoost model to allow it to extrapolate corrections to data that lie outside the training period. However, in the current setup, the feature importance of the time term is only significant at a few levels for some latitude bands (Figs. 1 and S1 to S4). This suggests that the time term does not play a major role in the model's predictions for these latitude bands. To improve the model's performance, we also tried to increase the number of trees, using Huber/quantile loss functions, but none of the changes helped to improve the significance of the time term. In summary, in the current setup, the time (date) term is not very important.

In Fig. 1, dynamical variables such as potential vorticity are most important in explaining the biases in the lower stratosphere (near 18 km). This is likely due to the fact that the TOMCAT model in the setup employed here overestimates the fast isentropic transport in the lower stratosphere. However, it is important to note that the 50–90∘ S region covers a large part of the high-latitude stratosphere and includes the strong wintertime polar vortex as well as tracer variations near the edge of the vortex. As a result, it is not possible to attribute the biases to a single variable or process. For example, temperature variations explain a large part of the CH4 biases around 35 to 40 km, but ClONO2 is most important just below 35 km. On the other hand, HNO3 is most important for explaining the N2O biases in the mid-upper stratosphere. This suggests that, while there is a strong relationship between temperature, potential vorticity and chlorine activation, the biases in CH4 and N2O at a single level are generally better explained by a single variable.

For CH4, additional features showing significant importances are water vapour, CH4 and N2O. As CH4 is the largest in situ source of stratospheric water vapour, their alternating importances in the lower mesosphere (above 55 km) indicate issues with HOx-related CH4 loss in the lower mesosphere. On the other hand, in the lower stratosphere, a strong wintertime dehydration inside the polar vortex leads to substantial drying. Hence, the somewhat larger importance for water vapour near 15 and 23 km suggests that XGBoost is able to identify and attribute possible biases in the TOMCAT setup to the downward transport of CH4 as well as the parameterized dehydration scheme. Similarly, the peaks in N2O importance near the stratopause (∼48 km) and near 21 km indicate issues in the representation of the downward transport of the long-lived tracers from the mesosphere into the stratosphere in the polar vortex. Note that, in our simulations, the TOMCAT top model level is located near 60 km.

Next we compare vertical CH4 profiles from TOMCAT, TCOM-CH4 and collocated HALOE/ACE for the SHpol latitude bin (Fig. 2). Overall, we have about 40 000 profiles, of which around 30 000 fall in the XGBoost training period and about 7000 profiles fall in the 2019–2021 evaluation period. Overall, TCOM-CH4 profiles show excellent agreement with satellite profiles, and median lines seem to follow each other very closely. In contrast, the TOMCAT profiles show good agreement with observational data between 20–30 km but exhibit positive biases at upper and lower levels. This distinct feature indicates a clear separation in the importance of dynamical and chemical processes controlling CH4 concentrations. As mentioned earlier, positive biases in TOMCAT CH4 in the lower stratosphere could be due to faster CH4 transport from the tropics to high latitudes. Positive biases in the upper stratosphere/lower mesosphere are most probably due to slower CH4 loss via HOx and ClOx chemistry. Another important characteristic of Fig. 2 is that the variability in observational profiles (the shaded regions show 10th to 90th percentile ranges) is much larger than that in the TOMCAT (or TCOM) profiles. A possible explanation for differences in variability would be that the model output is sampled at the longitude/latitude recorded at 30 km tangent height, but in reality collocations at different altitudes are a few degrees apart. Additionally, the onion-peeling algorithm used for some solar occultation measurements (such as SAGE, HALOE) assumes that observations at different tangent heights are independent; hence, retrieved profiles show larger fluctuations.

Vertical profiles of the absolute (in ppm) and percentage (%) CH4 differences between the three data sets are also shown in Fig. 2 for both the training (1992–2018) and evaluation (2019–2021) time periods. As expected, the median TCOM-CH4 profiles show very little difference with respect to collocated median satellite profiles, whereas the TOMCAT profile differences range from -0.22 ppm (16 km) to -0.05 (near to 28 km). In terms of relative differences, again TCOM–observation differences are close to 0 %, whereas for the evaluation period, differences are up to 10 % in the lower and middle stratosphere. A possible explanation for somewhat larger differences for the 2019–2021 time period is that there has been rapid increase in atmospheric CH4 over the last few years e.g.. As the period of rapidly increasing CH4 is outside the XGBoost training values, the estimated correction terms seem to be too small, but there are still significant improvements compared to TOMCAT profiles. Median profile comparisons for training and evaluation periods and subsequent differences (in ppm and %) for other latitude bins are shown in Figs. S5 to S8. Again, the TCOM–observation comparison is also consistent for other latitude bins, with an exception that mid-stratospheric biases for the evaluation period are somewhat larger (up to 10 %) for the SHmid and the tropics (Figs. S5 and S6).

Similarly, for N2O, Fig. 3 compares median profiles from ACE-FTS N2O, TCOM-N2O and TOMCAT and their differences (absolute and percentage) for SH polar latitudes. Again, the TCOM-N2O and ACE-FTS profiles show excellent agreement for all stratospheric altitudes. Interestingly, the TOMCAT N2O profiles are high-biased only in the lower stratosphere (up to 25 km); they have negligible biases in the mid-upper stratosphere. So, in the lower stratosphere, the TOMCAT profiles show positive biases of up to ±50 ppb, while TCOM-N2O biases are close to zero for the training period (2004–2010) but show a slight increase (up to ±10 ppb) for the evaluation period (2019–2021). Some of these biases could be linked to the use of measurements with positive values only, and it is possible that there is a missing variable that accounts for strong seasonal variations at higher latitudes in the current setup for XGBoost. Although TCOM-N2O biases are much smaller throughout the stratosphere, in percentage terms they can reach up to 100 % near to 40 km as changes in the small values can translate into much larger changes in relative differences. However, even with those large relative differences, significantly reduced biases in TCOM-N2O profiles are visible for all the levels.

Median profile comparisons and differences between ACE-FTS, TCOM-N2O and TOMCAT profiles (in ppb and %) for other latitude bins are shown in Figs. S9 to S12. Similar to SHpol, the absolute median differences between observed and TCOM values for other latitude bins are less than 10 ppb. However, the relative differences in the upper stratosphere are much larger (up to 100 %, especially in the SHmid and tropics). This is likely due to the fact that TCOM only uses positive values, which removes observed profiles with low concentration values during the winter months.

Improvements in CH4 and N2O profiles are clearly visible in the time series comparisons shown in Figs. 4 and 5, which compare CH4 and N2O evaluations at 20, 30, 40 and 50 km for the SHpol latitude bin. For clarity, the figure shows every 10th profile (10 % of the data points). Similar comparisons for SHmid, the tropics, NHmid and NHpol are shown in Figs. S13 to S20. TCOM-CH4 data points show excellent agreement with the HALOE and ACE data points (Fig. 4). Uneven data density before and after 2004 reflect differences in viewing techniques between these two satellite instruments. Basically, HALOE was designed to provide near-global coverage, whereas ACE-FTS was designed to provide denser coverage at high latitudes. Even with these uneven sampling frequencies, we do not observe any abrupt changes in TCOM-CH4 data points.

Similarly for N2O, Fig. 5 also shows excellent agreement between TCOM-N2O and ACE-FTS data points. Again, the largest corrections are observed in the lower stratosphere (15 to 25 km), where TOMCAT profiles are about 30 ppb high-biased, which can be considered a systematic bias due to the TOMCAT setup. Similar to CH4, a seasonal minimum occurs just after the break-up of the Antarctic polar vortex (October) as the descending branch of stratospheric circulation transports N2O-depleted air to lower altitudes and latitudes (horizontal mixing). As N2O mixing ratios decrease rapidly with increasing altitude, a large number of ACE-FTS data points show negligible N2O values, which is reflected in the TCOM-N2O data points. However, it is also important to note that both CH4 and N2O mixing ratios decrease rapidly with increasing altitude (especially during SH autumn/winter). As the forward model used in the ACE-FTS retrieval algorithm needs spectra at fixed height levels, a seasonal variation in the vertical structure of the atmosphere alters the spacing between tangent heights. Therefore, N2O (as well as CH4) profile variability increases when tangent heights get very close together. Additionally, as mixing ratio values get close to zero, retrieved profiles become noisy, as some values can be negative. Here, we use only positive data points for XGBoost training, so the correction terms used here might be positively biased, influencing the seasonal cycle effects on CH4 and N2O concentrations.

Another important feature seen in Figs. 4 and 5 is that the seasonal cycles in TCOM-CH4 and TCOM-N2O data points seem to be more synchronized with observational data sets than TOMCAT, especially at 20 km. As shown above, TOMCAT profiles show positive biases throughout the stratosphere, and the largest corrections seem to be in the summertime maximum values that must arise from transport from mid-high latitudes. Interestingly, near to 30 km, points from all three data sets seem to be closer to each other for both CH4 and N2O. Finally, an interesting aspect of both Figs. 4 and 5 is that in the upper stratosphere, both species show wintertime minima near to 40 to 50 km that are close to zero throughout the data record. Even with long-term trends in tropospheric concentrations, a casual inspection does not show any significant trends in either species. We aim to explore this aspect of CH4 and N2O trends in future studies.

Next, we compare TCOM-CH4 profiles with the latest SPARC CH4 data set . Figure 6 show daily mean zonal mean CH4 time series from TCOM-CH4 and monthly mean values from three SPARC (S-HALOE-CH4, S-MIPAS-CH4 and S-ACE-CH4) CH4 data records. Unsurprisingly, with some exceptions (near to 32.5∘ S and N), TCOM-CH4 shows the best agreement with S-ACE-CH4 data at all pressure levels and latitude bins. At 50 hPa, TCOM-CH4 values show somewhat positive biases with respect to S-HALOE-CH4 near subtropical latitudes but better agreement in the middle (5 hPa) and upper (0.5 hPa) stratosphere. On the other hand, S-MIPAS-CH4 data points show significant positive biases with respect to all other data records, with qualitative agreement in the upper stratosphere. Additionally, as expected, positive growth rates observed in the tropospheric CH4 concentrations are also distinguishable in both observations (ACE + HALOE) and TCOM-CH4 data, especially near tropical and subtropical latitudes at 50 hPa.

Figure 6 also compares the CH4 evolution at 67.5∘ S and 67.5∘ N. As expected, wintertime CH4 concentrations in the SH high latitudes are somewhat larger compared to those in the NH high latitudes e.g.. This is because a stronger Brewer–Dobson (BD) circulation in the NH causes faster downward propagation of the CH4-poor air from the upper stratosphere to the lower-middle stratosphere. Interestingly, all the SPARC CH4 data records show less CH4 in the SH at 50 hPa than TCOM. At 5 hPa, TCOM-CH4 data show better agreement with S-HALOE-CH4 data in both hemispheres and with S-ACE-CH4 data only in the NH. The exact causes of the unusually low CH4 values in S-MIPAS-CH4 and S-ACE-CH4 data files are unclear. A possible explanation might be that negative data points seen in ACE data (due to enhanced wintertime downwards transport of CH4-poor air) are excluded in the XGBoost training step. It also suggests that wintertime downward descent at higher latitudes is somewhat weaker in TCOM data. Again, S-MIPAS-CH4 data points indicate a much larger magnitude of the seasonal cycle compared to other data sets. In the upper stratosphere (0.5 hPa), there seems to better agreement among all the data in both hemispheres. Overall, we find that, compared to the TCOM-CH4 data set, SPARC CH4 data records have some inconsistent characteristics, and the largest disagreement is found to occur at NH high latitudes.

Figure 7 compares the evolution of TCOM-N2O and SPARC data sets based on MIPAS, Aura-MLS, SMR and ACE measurements for five latitude grids (67.5∘ S, 32.5∘ S, 2.5∘ N, 32.5∘ N and 67.5∘ N) and three pressure levels (50, 5 and 0.5 hPa). The lack of satellite measurements during the 1990s makes it difficult to compare the long-term N2O evolution, but significant differences between various satellite data records also complicate the more straightforward evaluation. Overall, TCOM-N2O shows the best agreement with SPARC ACE-FTS (S-ACE-N2O) data and the poorest agreement with SPARC MIPAS (S-MIPAS-N2O) data. Interestingly, SPARC SMR (S-SMR-N2O) shows N2O variations that are very similar to the S-MIPAS-N2O data set, whereas SPARC-Aura-MLS (S-AMLS-N2O) agrees better with S-ACE-N2O, with some exceptions in the later few years that are related to a drift in the MLS N2O measurement (190 GHz) channel , especially in the lower stratosphere. Hence, for the earlier period, TCOM-N2O also shows good agreement with S-AMLS-N2O data until 2014, and then slight drifts are distinguishable at low-mid latitudes. On the other hand, the close agreement between S-SMR-N2O and S-MIPAS-N2O means that both data sets exhibit high biases in the lower stratosphere, and again agreement is weakest at low-mid latitudes.

Another important aspect of Fig. 7 is that at high latitudes, as well as for mid-upper stratospheric altitudes, all the SPARC data sets agree quite well with each other, and there are no long-term drift and systematic biases between them. The good agreement of TCOM-N2O with all the SPARC N2O data sets at 5 and 0.5 hPa provides additional evidence of the usefulness of the TCOM-N2O data set. Additionally, similar to TCOM-CH4, casual inspection of TCOM-N2O does not show any long-term trends at mid-upper stratospheric pressure levels; we aim to investigate this further in future studies.

Next, we analyse differences between TCOM-CH4 and TOMCAT CH4 profiles through the time evolution of corrections estimated by the XGBoost regression model. First, we look at the differences in zonal mean CH4 at different levels. Figure 8 shows TCOM-CH4 minus TOMCAT CH4 differences (in %) at four vertical levels (15 to 45 km with 10 km spacing). An important aspect regarding 15 km and 25 km differences is that although the median CH4 differences shown in Fig. 2 indicate that TOMCAT profiles show positive biases (of up to 10 %), the latitude slice indicates significant variations between the two. Differences are even positive close to polar latitudes, indicating stronger downward transport of CH4-poor air and/or weaker mixing near the Antarctic polar vortex region in the TOMCAT simulation. Similar characteristics are observed at NH high latitudes. These biases are even more distinct at 25 km, especially in the SH high latitudes, though this region can be considered to be a boundary region where dynamical processes control CH4 concentrations at lower altitudes and chemical processes dominate at higher altitudes. Inter-hemispheric asymmetry in the CH4 bias correction also indicates significant differences in the representation of the BD circulation in ERA5 data e.g..

Additionally, some uneven differences for 1991–1993 at 15 and 25 km in Fig. 8 could be due to a combination of various chemical and dynamical processes. For example, volcanically enhanced stratospheric aerosol following the Mt Pinatubo eruption in June 1991 might have altered stratospheric transport pathways, as larger aerosols absorb outgoing long-wave radiation . Such heating can also enhance tropical upwelling as well as horizontal mixing on isentropic surfaces e.g.. Volcanically enhanced stratospheric aerosol can also alter OH radical concentrations either by modulating the amount of incoming solar radiation or by altering chemical pathways e.g.. It is also well known that increases in stratospheric aerosol concentration also affected HALOE retrievals e.g.. ERA5 data assimilation does not include these atmospheric effects of volcanically enhanced stratospheric aerosol e.g.; hence, we are not sure about the unusual CH4 differences in the lower stratosphere.

On the other hand, differences at 35 km in Fig. 8 seem to be dominated by the quasi-biennial oscillation (QBO). QBO-induced meridional circulation patterns e.g.,, which are underestimated in TOMCAT. Even though ACE provides limited observational data points in the tropics, XGBoost is able to identify this discrepancy. On the temporal scale, differences are largest until 1996, reaching polar latitudes, after which they gradually decrease in the NH sub-tropics and remain larger in the SH sub-tropics. A similar type of uneven evolution for later periods can also be seen, suggesting issues with the ERA5 data regarding the representation of QBO-induced circulation, especially for years with unusual QBO evolution such as 2016 and 2020 e.g..

Another notable feature in Fig. 8 is that at 45 km, CH4 differences are clearly distinguishable in some years. Both HALOE and ACE have much smaller retrieval errors at higher altitudes, and, assuming there were no abrupt changes in these two satellite instruments (or retrieval algorithms), the unusual differences seen at 45 km can be attributed to inhomogeneities in or issues with the ERA5 data. These distinctive periods include the first halves of years 1993, 1997, 2001 and 2004 and the latter half of 2019.

HALOE V19 data are available from https://acdisc.gesdisc.eosdis.nasa.gov/data//UARS_HALOE_Level2/ . ACE-FTS v4.2 data are obtained via http://www.ace.uwaterloo.ca/data.php (last access: 13 January 2022) (DOI: 10.20383/101.0291, ). SPARC climatological data can be obtained via 10.5281/zenodo.4265393 . TCOM-CH4 and TCO-N2O data are publicly available at 10.5281/zenodo.7293740 and 10.5281/zenodo.7386001 , respectively.

Even though CH4 and N2O are very important greenhouse gases, as well as the sources for key stratospheric species, there are limited stratospheric profile data sets that extend for more than a decade. Due to their long lifetimes, CH4 and N2O are also very useful dynamical tracers that can used to evaluate stratospheric transport processes. Also, accurate stratospheric CH4 profiles are a valuable constraint for the retrieval of tropospheric methane using satellite instruments. However, until now, no attempt has been made to construct long-term CH4 and N2O profile data sets. Furthermore, although chemical models are able to simulate long-term profile data sets of these species, they are highly dependent on the representation of individual chemical and dynamical processes in a particular model.

Here we have used CH4 and N2O profiles from the TOMCAT CTM, two solar occultation instrument measurements, and the eXtreme Gradient Boost (XGBoost) regression model to construct daily, gap-free stratospheric profile data sets (TCOM-CH4 and TCOM-N2O) for the 1991–2021 time period. The XGBoost regression model is trained for the CH4 or N2O difference between TOMCAT and satellite measurements (HALOE and ACE). These differences are used to estimate corrections that are added to the TOMCAT profiles to derive TCOM-CH4 and TCOM-N2O profiles. The regression algorithm uses 13 features (or variables) based on TOMCAT tracers as well as four additional features such as temperature, potential vorticity, latitude and date of the measurement. Because atmospheric concentrations of CH4 and N2O vary due to distinct dynamical and chemical processes in different regions, our approach involves dividing global measurements into five latitude-based categories. These categories include two for the polar regions, two for the mid-latitudes, and one for the tropics. We then proceed to derive regression parameters for each 1 km vertical grid spanning from 15 to 60 km within each of these latitude bins.

For both gases considered, XGBoost shows good performance (R2>0.5 to 0.8) throughout the stratosphere except for the lower stratosphere, which can be attributed to the limited training measurements. Measurements from the final 3 years (2019–2021) are used to evaluate TCOM-CH4 and TCOM-N2O profiles. Overall, TCOM-CH4 shows excellent agreement with the evaluation profiles, and median differences are less than 10 %. Additionally, comparison with SPARC-CH4 data suggests that SPARC-MIPAS profiles show some unrealistic behaviour, and SPARC-ACE and SPARC-HALOE do not show expected inter-hemispheric differences in lower stratospheric CH4 differences (there is less CH4 in the NH).

For TCOM-N2O, better agreement is again seen with respect to the S-ACE-N2O data set and weaker agreement is observed with MIPAS data. TCOM-N2O also confirms the abnormal drift in the Aura-MLS v4.2 N2O data (as used in the SPARC data set), especially at lower latitudes and altitudes e.g.. A casual inspection of TCOM-CH4 and TCOM-N2O plots also suggests that, despite increasing surface values, there are near-negligible long-term trends in the upper stratosphere/lower mesosphere, which is consistent with . On the other hand, analysed MLS V5 data and showed positive trends (of up to 15 %) in the tropical upper stratospheric N2O, though they did not find NO production to be rising at similar rates. A possible explanation would be that stratospheric/mesospheric loss processes, probably caused by changes in the stratospheric circulation, are reducing the lifetimes of these GHGs. We aim to analyse these discrepancies in future studies. Finally, analysis of TCOM-CH4 and TOMCAT CH4 profiles suggests that the representation of QBO-induced secondary circulation is not adequate in the CTM, and differences also reveal some temporal inhomogeneities in the ERA5 reanalysis data.

Presently, the TCOM-CH4 and TCOM-N2O V1.0 data set is ideally suited for the evaluation of CH4 and N2O chemistry and stratospheric transport processes in models. The TCOM-CH4 data set can also be used as realistic stratospheric profiles in a CH4 profile/total column retrievals. Daily mean zonal mean TCOM-CH4 and TCOM-N2O profile data on pressure and altitude levels in mixing ratio units are publicly available via 10.5281/zenodo.7293740 and 10.5281/zenodo.7386001 , respectively.