The MUSICA IASI full product data are provided as netCDF files compliant with version 1.7 of the CF (Climate and Forecast) metadata convention (https://cfconventions.org, last access: 25 January 2022​​​​​​​). The data files contain all information needed for reproducing the retrievals and for optimally reusing the data. Because the MUSICA IASI retrieval builds upon the EUMETSAT L2 cloud filter and uses the EUMETSAT L2 atmospheric temperature as the a priori atmospheric temperature, the output files contain some EUMETSAT retrieval data as well as the MUSICA retrieval data. In addition, they contain the EUMETSAT L1C spectral radiances (and the simulated radiances) as well as auxiliary data needed for the retrieval (like surface emissivity from other sources; ).

We provide standard output files comprising all processed IASI observations and one extended output file with detailed calculations of Jacobians (and spectral responses for surface emissivity, spectroscopic parameters, and cloud coverage) and gain matrices for a few selected observations.

The standard output is provided in the files “IASI[S]_MUSICA_[V]_L2_AllTargetProducts_[D]_[O].nc” and in one file per orbit and instrument. The symbols within the square brackets indicate placeholders: “[S]” for the sensor (A, B, or C, for IASI instruments on the satellites Metop-A, Metop-B, or Metop-C, respectively), “[V]” for the MUSICA IASI retrieval processor version used, “[D]” for the starting date and time of the observation (format YYYYMMDDhhmmss), and “[O]” for the number of the orbit.

In our database these files are provided in daily .tar files, with all orbits of all IASI instruments archived into a single .tar file, with the name “IASI[multipleS]_MUSICA_[V]_L2_AllTargetProducts_[DAY].tar”. The placeholders are as follows: “[multipleS]” for the considered sensors, e.g. AB if IASI sensors on Metop-A and Metop-B are considered; “[V]” for the MUSICA IASI retrieval processor version used; and “[DAY]” for the date of observations (universal time, format YYYYMMDD). The typical size of a .tar file with the orbit-wise netCDF files of a single day is 15 GB. This number is for the typically 28 orbits per day of two satellites (for three satellites there are typically 42 orbits per day). The standard output data files are linked to a DOI .

The extended file represents 74 observations over polar, mid-latitudinal, and tropical GRUAN stations (GRUAN stands for Global Climate Observing System Reference Upper-Air Network, https://www.gruan.org, last access: 25 January 2022​​​​​​​). More details on the time periods and locations represented by these retrievals are given in . The file provides the same output as the standard files and in addition detailed information on Jacobians (and spectral responses) and gain matrices. The Jacobian matrices collect the derivatives of the radiances as measured by the satellite sensors with respect to a parameter (e.g. atmospheric temperature, instrumental conditions). The spectral response matrices give information about the change in the radiances due to changes in the surface emissivities, the spectroscopic parameters, and the cloud coverage. The gain matrices are the derivatives of the retrieved atmospheric state with respect to the radiances. The name of this extended output file is “IASIAB_MUSICA_030201_L2_AllTargetProductsExtended_examples.nc”; its size is 70 MB, and it is linked to an extra DOI .

Here we report on the MUSICA IASI processing version 3.2.1 (applied for IASI observations until the end of June 2019). For observations from July 2019 onward processing versions 3.3.0 and 3.3.1 are applied (the difference of version 3.3.1 with respect to 3.3.0 is updated a priori data for the retrievals of observations from January 2021 onward). Version 3.2.1 and the versions 3.3.x use the same retrieval setting, and the output files contain the same variables. The difference between version 3.2.1 and the versions 3.3.x is that for the former some minor corrections were necessary after the retrieval process due to some very minor inconsistencies in version 3.2.1 with regard to the following: the vertical gridding; the a priori of δD; and the constraint for N2O, CH4, and HNO3. This difference between the versions is actually not noticeable by the user, and the report provided here on version 3.2.1 data is also valid for data of versions 3.3.x, which will soon be made available for the public in the same format as the version 3.2.1 data.

There are three different categories of variables. The first category consists of variables that contain information resulting from the EUMETSAT L2 PPF (product processing facility) retrieval. They can be identified by the prefix eumetsat_ in their names. A second category consists of variables that contain information from the MUSICA IASI retrieval. Here the prefix in the name is musica_. The third category encompasses all other variables, and their names have no specific prefix.

The EUMETSAT L2 retrieval variables are flags (mainly for cloud coverage – see Sect. , surface conditions, and EUMETSAT retrieval quality) and the EUMETSAT L2 retrieval output of H2O. The variables belonging to the third category are supporting data and inform about the sensors' viewing geometry, observation time, measured radiances, climatological tropopause altitude, and surface emissivity. Although our MUSICA IASI retrieval uses the EUMETSAT L2 PPF version 6 land surface emissivity, the emissivity variables are assigned to the category of supporting data, because for older observations where no L2 PPF version 6 is available we use the surface emissivity climatology from IREMIS and over water we always use the values reported by . The large majority of variables are MUSICA IASI variables. These variables document the MUSICA IASI retrieval settings (like the a priori states and constraints; see Sect.  to ), provide the MUSICA IASI retrieval products (retrieved trace gas profiles, Sect. ), and characterise these products (averaging kernels, estimated errors, Sect. ). For variables that refer to a specific retrieval product, a corresponding syllable is embedded into the respective variable names: _wv_ and _wvp_ stand for water vapour isotopologues and water vapour isotopologue proxies, respectively; _ghg_ for the greenhouse gases N2O and CH4, _hno3_ for HNO3; and _at_ for the atmospheric temperature.

The water vapour and greenhouse gas variables (_wv_, _wvp_, and _ghg_) contain information on two species, which can be identified by the value of the dimension musica_species_id. For _wv_ these are the species H2O and HDO, for _wvp_ the water vapour proxy species (see Sect. ), and for _ghg_ the species N2O and CH4.

We focus on the processing of IASI data for which EUMETSAT L2 data files of PPF version 6.0 or later are available. For former data versions not all of the subsequently discussed L2 PPF variables are available. Furthermore, we found that there are several modifications made within versions 4 and 5 that significantly affect the stability of our MUSICA IASI retrieval output (see discussion in ). EUMETSAT L2 PPF version 6 data are available from October 2014 onward, so we focus our processing on IASI observations made from October 2014 onward.

In addition, the MUSICA IASI retrievals are currently restricted to cloud-free scenarios. The selection of cloud-free conditions is made by means of the EUMETSAT L2 PPF cloudiness assessment summary flag variable (called flag_cldnes in the EUMETSAT L2 netCDF data files). We only process IASI observations with this flag having the value 1 (the IASI instrumental field of view, IFOV, is clear) or 2 (the IASI IFOV is processed as cloud-free, but small cloud contamination is possible). This requirement for cloud-free scenarios removes more than two-thirds of all available IASI observations.

Furthermore, we require EUMETSAT L2 PPF temperature profiles to be generated by the EUMETSAT L2 PPF optimal estimation retrieval scheme. For this purpose we use the EUMETSAT L2 PPF variable flag_itconv. We only process data with this flag having value 3 (the minimisation did not converge, sounding accepted) or 5 (the minimisation converged, sounding accepted).

Figure  gives a climatological overview of the number of IASI data that remain after the aforementioned preselection. The maps largely reflect the cloud cover conditions. A very large number of IASI data passed our selection criteria in the subtropical regions, where cloud-free conditions generally prevail. In the North Atlantic storm track region, the southern American and southern African tropics, and the southern polar oceans the sky is generally cloudy in February, leading to a low number of IASI observations that passed our selection criteria. In August we can clearly identify the Asian and West African monsoon region as an area with increased cloud coverage and consequently fewer MUSICA IASI processed data.

Figure  is similar to Fig. , but instead of showing the total number of observations that fall within a 1∘×1∘ box, it depicts the probability of having at least one observation per overpass in a 1∘×1∘ box. In both figures we observe very similar structures.

We use the thermal nadir retrieval algorithm PROFFIT-nadir . It is an extension of the PROFFIT algorithm (PROFile FIT; ) that has been used for many years by the ground-based infrared remote sensing community . This extension has been made in support of the IASI retrieval development during the project MUSICA. The algorithm consists of the line-by-line radiative transfer code PROFFWD and can consider Voigt as well as non-Voigt line shapes and the water continuum signatures according to the model MT_CKD v2.5.2 . For the MUSICA IASI processing we use the water continuum model MT_CKD v2.5.2 and for all trace gases a Voigt line shape model and the spectroscopic line parameters according to the HITRAN2016 molecular spectroscopic database . However, we increase the line intensity parameter for all HDO lines by +10 % in order to correct for the bias observed between MUSICA IASI δD retrievals and respective aircraft-based in situ profile data .

For the inversion calculations PROFFIT-nadir offers options that are essential for water vapour isotopologue retrievals. These are the options for logarithmic-scale retrievals and for setting up a cross constraint between different atmospheric species (see also Sect. ). The theoretical basics for atmospheric trace gas retrievals are provided in Appendix .

The retrieval works with the radiances measured in the spectral region between 1190 and 1400 cm-1. The respective radiance values are the elements of the MUSICA IASI measurement state vector referred to as y in Appendix . Figure  depicts measured and simulated radiances as well as a large variety of different spectral responses (Jacobians multiplied by parameter changes) for a typical mid-latitudinal summer observation over land. Please note the different radiance scales for measurement and simulation, on the one hand, and residuals and spectral responses, on the other hand.

We show trace gas spectral responses for a uniform increase in the trace gases throughout the whole atmosphere: 100 % for H2O and HDO, 10 % for N2O and CH4, and 50 % for HNO3. The respective values are reasonable approximations of the typical atmospheric variabilities of these trace gases. We see that the measured radiances are most strongly affected by the water isotopologues. The variations in N2O and CH4 are also recognisable (larger than the spectral residuals, i.e. the difference between measured and simulated radiances). The spectral responses of HNO3 are very close to the noise level.

The atmospheric temperature spectral responses are depicted for a uniform 2 K temperature increase over three different layers: surface–2, 2–6, and 6–12 km a.s.l. (a.s.l. means above sea level). In the analysed spectral region (1190–1400 cm-1), the atmospheric temperature variations close to the surface affect mainly the radiances below 1300 cm-1 and variations at higher altitudes mainly the radiances above 1300 cm-1. In Fig.  we depict the spectral responses for 2 K because this is a reasonable approximation of the uncertainty in the EUMETSAT L2 PPF temperatures .

The spectral responses for surface emissivity and temperature reveal that surface properties hardly affect the radiances above 1250 cm-1 but have a strong impact below 1250 cm-1. We calculate the emissivity spectral responses for a -2 % change in the emissivity independently above and below 1270 -1, which is a typical uncertainty in emissivity judging from its dependency on the viewing angle and wind speed over ocean and small-scale inhomogeneities; however, this uncertainty might be significantly higher over arid areas .

Concerning spectroscopy, the spectral responses calculated for the typical uncertainty range of spectroscopic parameters are relatively small. In Fig.  we show the spectral responses for consistent +5 % changes in the line intensity and pressure-broadening parameters of all water vapour isotopologues, which are in reasonable agreement with the uncertainty values given by HITRAN . Concerning the water continuum, the spectral responses are for a water continuum that is 10 % larger than the continuum according to the model MT_CKD v2.5.2 .

The bottom panel of Fig.  depicts the impact of clouds on the radiances. The thermal nadir radiance when observing over an opaque cloud can be calculated by defining the cloud top instead of the surface as the thermal background. Cirrus and mineral dust clouds are not opaque, and we have to consider partial attenuation by the cloud particles. We calculate the attenuated radiances using forward model calculations from KOPRA (Karlsruhe Optimized and Precise Radiative transfer Algorithm; ) and consider single scattering. The frequency dependency of the extinction cross sections, the single-scattering albedo, and the scattering phase functions of the clouds are calculated from OPAC v4.0b (Optical Properties of Aerosol and Clouds; ). For cirrus clouds we assume the particle composition as given by OPAC's “Cirrus 3” ice cloud example (see Table 1b in ) and for mineral dust clouds a particle composition according to OPAC's “Desert” aerosol composition example (see Table 4 in ). The spectral responses shown are for 10 % cumulus cloud coverage with the cloud top at 3 km, a homogeneous dust cloud between 2 and 4 km, and 25 % cirrus cloud coverage between 10 and 11 km. These are relatively weak clouds, and we assume that they might occasionally not correctly be identified by the EUMETSAT L2 cloud screening algorithm. Because the respective spectral responses are significantly above the noise level, these unrecognised clouds can have an important impact on the retrieval.

A comprehensive set of different spectral responses is provided with the extended output data file for the 74 exemplary observations at an Arctic, mid-latitudinal, and tropical site.

In this section we discuss the MUSICA IASI state vector, which is referred to as x in Appendix .

The reference for the a priori data used for the MUSICA IASI trace gas retrievals is the CESM1–WACCM (Community Earth System Model version 1 – Whole Atmosphere Community Climate Model) monthly output of the 1979–2014 time period. The CESM1–WACCM is a coupled chemistry climate model from the Earth's surface to the lower thermosphere . The horizontal resolution is 1.9∘ latitude × 2.5∘ longitude. The vertical resolution in the lower stratosphere ranges from 1.2 km near the tropopause to about 2 km near the stratopause; in the mesosphere and thermosphere the vertical resolution is about 3 km. Simulations used for generating the MUSICA IASI a priori data are based on the International Global Atmospheric Chemistry–Stratosphere-troposphere Processes And their Role in Climate (IGAC–SPARC) Chemistry Climate Model Initiative (CCMI; ). From the surface to 50 km the meteorological fields are “nudged” towards meteorological analysis taken from the National Aeronautics and Space Administration (NASA) Global Modeling and Assimilation Office (GMAO) Modern-Era Retrospective Analysis for Research and Applications (MERRA; ), and above 60 km the model meteorological fields are fully interactive, with a linear transition in between (details about the nudging approach are described in ).

For the MUSICA IASI a priori profiles of H2O, N2O, CH4, and HNO3, we consider a mean latitudinal dependence, seasonal cycles, and long-term evolution. Therefore, the a priori data are constructed by means of a low dimensional multi-regression fit on the CESM1–WACCM data independently for each vertical grid level. We fit an annual cycle with the two frequencies 1 per year and 2 per year, and for the long-term baseline we fit a second-order polynomial. The fits are performed individually for 15 equidistant latitudinal bands between 90∘ S and 90∘ N. In order to capture the yearly anomalies in N2O and CH4 a priori data, we use the Mauna Loa Global Atmospheric Watch yearly mean data records for a correction of the WACCM parameterised time series (for more details on this correction procedure see ). We also use the temperature lapse rate tropopause – according to the definition of the World Meteorological Organization – from WACCM and construct a latitudinally dependent tropopause altitude by fitting a seasonal cycle and a constant baseline (no long-term dependency) and assume a transition zone between the troposphere and stratosphere with a vertical extension of 12.5 km. The MUSICA IASI δD a priori profiles between the ground and the tropopause altitude are constructed from the H2O a priori profiles by using a single global relation between tropospheric H2O concentration and δD values. This relation has been determined from simultaneous H2O and δD measurements made by high-precision in situ instruments at different ground stations located in the mid-latitudes and the subtropics and between 100 m and 3650 m a.s.l.  and by aircraft-based in situ measurements made between the sea surface and about 7000 m a.s.l. . Above the troposphere (where δD is close to -600 ‰) we smoothly connect the tropospheric δD values with the typical stratospheric δD value of -350 ‰.

Figure  depicts the MUSICA IASI a priori data derived from WACCM. It shows latitudinal cross sections for a northern hemispheric winter and summer day as well as the temporal evolution between 2014 and 2020 at a mid-latitudinal site. The H2O and δD a priori data have strong latitudinal gradients and also a marked seasonal cycle. For δD the lowest values are in the neighbourhood of the tropopause altitude (depicted as a thick violet line). The a priori values of N2O and CH4 have a strong latitudinal and seasonal variability in the tropopause region. CH4 has a strong tropospheric latitudinal gradient and seasonal cycle in the troposphere, whereas the tropospheric N2O variability is rather small. The HNO3 a priori has a maximum in the lower stratosphere (20–25 km) with the highest values at higher latitudes.

The a priori trace gas profiles are provided in the variables musica_wv_apriori (H2O and HDO with species index 1 and 2, respectively), musica_ghg_apriori (N2O and CH4 with species index 1 and 2, respectively), and musica_hno3_apriori (HNO3). The unit is ppmv.

As a priori for the atmospheric and the surface temperatures we use the EUMETSAT L2 PPF atmospheric temperature output. These data are provided in the unit kelvin and in the variables musica_at_apriori and musica_st_apriori for atmospheric temperature and surface temperature, respectively.

We set up simplified a priori covariance matrices by means of two parameters. The first parameter is the altitude-dependent amplitudes of the variability (vamp,i, with i indexing the ith altitude level). For the trace gases we work with the relative variability, i.e. with the variability on the logarithmic scale. For atmospheric temperatures the variability is given in the unit kelvin. The second parameter is the altitude-dependent vertical correlation lengths (σcl,i, for considering correlated variations between different altitudes). The elements of the a priori covariance matrix (Sa​​​​​​​) are then calculated as 7Sai,j=vamp,ivamp,jexp⁡-(zi-zj)22σcl,iσcl,j, with zi being the altitude at the ith altitude level.

The values vamp,i and σcl,i are oriented to the typical covariances of in situ observations made from the ground (e.g. ), aircraft (e.g. ), or balloons (e.g. ) and also aligned to the vertical dependency of the monthly mean covariances we obtain from the WACCM simulations. For the vamp,i of δD we use in addition the isotopologue-enabled version of the Laboratoire de Météorologie Dynamique (LMD) general circulation model as a reference . For atmospheric temperature we use the uncertainty in the EUMETSAT L2 atmospheric temperature as reference . Generally, we classify three different altitude regions with specific vertical dependencies in the values of vamp,i and σcl,i: the troposphere (below the climatological tropopause altitude as depicted in Fig. ), the stratosphere (starting 12.5 km above the climatological tropopause altitude), and the transition region between the troposphere and stratosphere.

The values of vamp,i are specific for each trace gas and for the atmospheric temperature, and they are provided in the MUSICA IASI standard output files in the variables having the suffix _apriori_amp.

As a simplification we use the same values of σcl,i for all trace gases and for the atmospheric temperature. These values are provided in the MUSICA IASI output files as the variable musica_apriori_cl.

As the constraint of the retrieval we use an approximation of the inverse of the covariance matrix. For this purpose the constraint matrix R is constructed as a sum of a diagonal constraint, and first- and second-order Tikhonov-type regularisation matrices : 8R=(α0L0)Tα0L0+(α1L1)Tα1L1+(α2L2)Tα2L2, with 9L0=100⋯0010⋯0001⋯0⋮⋮⋮⋱⋮000⋯1,10L1=1-10⋯001-1⋯0⋮⋮⋱⋱⋮0⋯01-1, and 11L2=1-21⋯0⋮⋱⋱⋱⋮0⋯1-21. The diagonal elements of the diagonal matrices α0, α1, and α2 are the inverse of the absolute variabilities and the variabilities of the first and the second vertical derivatives of the profiles. These values can be calculated from the elements of the a priori matrix (Sa) as follows: 12α0i,i=(Sai,i)-1,13α1i,i=(Sai,i+Sai+1,i+1-2Sai,i+1)-1,14α2i,i=Sai,i+4Sai+1,i+1+Sai+2,i+2-4Sai,i+1-2Sai,i+2+4Sai+1,i+2-1. Starting the retrievals with the constraint matrix R≈Sa-1 optimises the computational efficiency of the retrieval processes because according to Eqs. () and () the retrieval calculations work with Sa-1. Furthermore, calculating the inversion of Sa approximatively as the sum of diagonal constraint and first- and second-order Tikhonov-type regularisation matrices offers the possibility of tuning the constraint according to specific user requirements with respect to smoothness or absolute deviations (e.g. ).

For the greenhouse gases (N2O and CH4) and HNO3 we constrain with respect to the absolute values of the profiles and the first derivative of the profile; i.e. we do not consider the term (α2L2)Tα2L2 of Eq. (). In the case of the water vapour isotopologue proxies and the atmospheric temperature, we additionally constrain with respect to the second derivative of the profile; i.e. we consider all terms of Eq. (). Please note that for the trace gases the constraints work on the logarithmic scale and for the atmospheric temperature on the linear scale.

Because HNO3 has only very weak spectroscopic signatures in the analysed spectral region (see Fig. ), we loosen the absolute constraint and at the same time strengthen the constraint with respect to the first vertical derivate: α0 and α1 are calculated from an Sa constructed with the values of vamp,i increased by a factor of 1.5 and with the values of σcl,i increased by a factor of 2. Similarly and in order to avoid a negative impact of an underconstrained retrieval of the temperature profile on the trace gas products (e.g. artificial oscillatory features), we strengthen the atmospheric temperature constraint: α0, α1, and α2 are calculated from an Sa constructed with the values of vamp,i decreased by a factor of 0.5.

The diagonal entries of the diagonal matrices α0, α1, and α2 contain all information about the actual constraints used by the retrieval. They are provided in the MUSICA output files for each individual retrieval and for the different trace gases and the atmospheric temperature as variables with the suffix _reg. For the trace gases these vector elements are depicted in Fig.  for a northern hemispheric summer in the tropics, mid-latitudes, and polar regions. The dotted lines indicate the climatological tropopause and the altitude 12.5 km above this tropopause (transition zone between the troposphere and stratosphere).

The retrieved trace gas profiles are provided in the variables musica_wv (H2O and HDO with species index 1 and 2, respectively), musica_ghg (N2O and CH4 with species index 1 and 2, respectively), and musica_hno3 (HNO3). The unit is ppmv. The retrieved atmospheric temperature is provided in the variable musica_at and the retrieved surface temperature in the variable musica_st. The unit is kelvin.

In order to provide a brief insight into the data diversity, Fig.  gives examples with a priori and retrieved trace gas profiles for an observation on 30 August 2008 over Lindenberg (53∘ N). The profile data represent 28 altitude levels and are provided with detailed information on their sensitivity, vertical representativeness, and errors (see following subsections).

For a limited number of retrievals we provide an extended netCDF output file (see Sect. ). The extended output file contains the same variables as the standard output files and in addition the full averaging kernels and a large set of Jacobians (and spectral responses for surface emissivity, spectroscopic parameters, and cloud coverage) together with gain matrices. The latter allows the calculation of full error covariances for a large variety of different uncertainty sources. In the standard output files we do not provide the full averaging kernels (which would consider all the cross-correlations between the different retrieval products) or the full error covariances. The reason for this is that providing the full kernels and/or the full error covariances would strongly increase the storage needs for the data output .

Figure  explains the matrix blocks that are made available in the extended output file and in all standard output files. The extended file contains the full gain matrices, the Jacobian matrices for all state vector components, and Jacobians for parameters that are not retrieved but that affect the retrieval (spectroscopy, different cloud types, and surface emissivity). Using the gain matrices and the Jacobians, the full averaging kernels and the full error covariances can be calculated as indicated by Fig. . The full averaging kernel for the trace gas products is marked at the right side by the thick black frame (an example for these kernels is plotted in Fig. ). The full error covariances are indicated by the yellow frame (examples of the root-mean-square values of the diagonals of these error covariances are plotted in Fig. ).

The parts of this full matrix that are provided by the standard output files for all individual retrievals are indicated as the matrix blocks filled by green and red colour. Green represents the individual averaging kernels of the water vapour isotopologues, the greenhouse gases, HNO3, and the atmospheric temperature. Red marks the cross kernels of the trace gas products with respect to atmospheric temperature (i.e. they indicate how errors in the EUMETSAT L2 PPF atmospheric temperatures – used as MUSICA IASI a priori temperatures – affect the retrieved trace gas products). These temperature cross kernels allow the calculation of the full error covariances for the temperature uncertainty for each individual observation of the standard output file.

In addition, for all individual observations the standard output files contain square root values of the diagonal of the error covariance matrix for the most important uncertainty sources (noise and temperature uncertainty).

We always provide differential or derivatives (covariances, averaging kernels, gain matrices, and Jacobian matrices) related to the trace gas products in the logarithmic scale. Logarithmic-scale kernels are the same as the fractional kernels used in . Furthermore, we strongly recommend the use of the logarithmic-scale kernels for analytic calculation. Because the MUSICA IASI trace gas retrievals are made on the logarithmic scale, the assumption of a moderately non-linear case according to can be made on the logarithmic scale (i.e. requires the use of logarithmic-scale kernels) but has limited validity on the linear scale. More details on the valid assumption of moderately non-linear problems are given in Appendix .

The EUMETSAT L2 PPF flag flag_cldnes is written in the MUSICA IASI variable eumetsat_cloud_summary_flag. As discussed in Sect.  there is some risk that the MUSICA IASI product retrieved for eumetsat_cloud_summary_flag set to 2 has significant errors due to clouds. In order to exclude this risk we can filter out these data; i.e. we can use a very stringent cloud filtering criterion by using only observations where the variable eumetsat_cloud_summary_flag is set to 1.

Another and less stringent option is to use in addition the EUMETSAT L2 fractional cloud cover, which is written in the MUSICA IASI variable eumetsat_cloud_area_fraction. If eumetsat_cloud_summary_flag is set to 2 we require in addition that the determination of a cloud area fraction has not been successful; i.e. we require that eumetsat_cloud_area_fraction is set to NaN. No clear determination of a value for fractional cloud cover means that the cloud signals are rather weak (the contrast between cloud and surface signals is smaller than the instrument noise).

The spectral noise level considered in the cost function Eq. () during the MUSICA IASI processing is the root-mean square (rms) of the spectral fit residual (difference between the simulated and measured spectrum). By this retrieval setting we use the degree to which the spectra can be understood by the forward model as the spectral noise level. The so-defined spectral noise level is generally larger than the pure instrumental noise level because it is a sum of the instrumental noise and the signatures that are not understood by the forward model. In the MUSICA IASI retrieval this rms value is treated as white noise; i.e. for Sy,noise of the cost function Eq. () we use a diagonal matrix filled by the mean-square values according to the spectral residuals.

As long as the residual is close to white noise, this kind of processing ensures a correct weighting of the measured spectra, on the one hand, and the a priori information, on the other hand. However, occasionally the measured spectra are very poorly simulated by the forward model and the residuals cannot be described as white noise; instead the residuals show systematic signatures. This happens, for instance, if incorrect surface emissivities are used or if the retrieval is made for an observation that is affected by a cloud. In order to identify the systematic part of the residuals we smooth the residuals using a ±2 cm-1 running mean. The smoothed residuals are the systematic residuals, and the difference between the original residuals and the smoothed residuals can then be interpreted as the random (or white noise) residuals. Residuals, systematic residuals, and random residuals are provided in the standard files for each observation in the variable musica_fit_quality.

In order to facilitate the filtering of data corresponding to a poor spectral fit quality, we set up a flag (provided as variable musica_fit_quality_flag) that works with the rms values of the systematic residuals and the random residuals. The flag is set to 0 (poor quality) if the systematic residuals have an rms value of larger than 40 nW/(cm2 sr cm-1). For all other observations we analyse the ratio between the rms of the systematic residuals and the rms of the random residuals. If this ratio is larger than 1.0, the flag is set to 1 (restricted quality); if it is between 0.5 and 1.0, the flag is set to 2 (fair quality); and if it is smaller than or equal to 0.5, the flag is set to 3 (good quality). Figure  depicts residuals corresponding to different values of this fit quality flag. All observations are made during the same orbit, at close-by locations (northern Africa), and for very similar surface temperatures. It is very likely that the poor spectral fit quality is due to incorrect surface emissivity values used for the respective retrievals (over arid areas like northern Africa, surface emissivity data have an increased uncertainty; ). Our recommendation is to use data that belong to the quality groups fair and good.

For all observations and all trace gas products, the standard files provide estimations of the errors dominating the random-error budget: errors due to noise in the spectra and errors due to uncertainties in the atmospheric temperature a priori data (the EUMETSAT L2 PPF temperatures). The noise error and estimations of atmospheric temperature error are given in the error variable (variable with suffix _error; see Sect. ) for all trace gas products and can be used for filtering out data with anomalously high errors.

Incorrect spectroscopic parameters (line intensity, pressure-broadening coefficients, or water continuum modelling) can be responsible for large errors. Although these uncertainty sources are systematic, the errors they cause depend on the sensitivity of the remote sensing system, which in turn is affected by the geometry of the observation. In the first order the optical path of the measured radiances depends on the platform zenith angle (PZA, provided as the variable platform_zenith_angle). In order to avoid that systematic uncertainties in the spectroscopic parameters cause artificial signals, we can set threshold values for the PZA and limit the PZA to angles close to nadir (e.g. by requiring PZA≤30∘).

The standard files provide the averaging kernels in a compressed format for all observations (see Sect. ) as well as metrics that capture the most important aspects of the sensitivity and vertical resolution (see Sect. ). These metrics are provided in the variables with the suffixes _response and _resolution and allow analyses of the sensitivity and vertical resolution for each individual observation without the need for reconstructing the averaging kernels. We can use the metrics for filtering out data where the response to the real atmospheric variability is low or where the vertical representativeness is irregular.

In order to ensure a good sensitivity (retrieval product being mainly affected by the real atmosphere and not by the a priori assumption), the measurement response (MR) should be close to unity. Layer width per DOFS (LWpD), centre altitude displacement (C-Alt​​​​​​​), and resolving length (RL) can be used to filter out data that do not fulfil the requirements in terms of the vertical representativeness needed for a dedicated study. Respective filter threshold values depend on the objective of the scientific study. If processes within vertically well confined layers shall be examined, rather small vertical displacement and very good vertical resolution are required, and thus very stringent thresholds should be set.

In addition to filtering according to absolute values of LWpD, C-Alt​​​​​​​, or RL, the respective metrics can also be used for the identification of groups of data that have a similar vertical representativeness. For instance, we can robustly analyse time series of data that have a stable vertical information displacement and a stable vertical resolution. For data where these conditions are not fulfilled, time series signals might be significantly affected by the time-variant data characteristics. The same is true when analysing horizontal patterns, which might partly be due to the pattern in the data characteristics and not a real atmospheric pattern.

We filter the data according to the settings and threshold values of Table . For all data we require “fair” and “good” for the MUSICA IASI spectral fit quality (flag variable musica_fit_quality_flag is required to be set to 2 or 3), and we filter the data using the EUMETSAT L2 PPF cloudiness assessment flag (provided as the variable eumetsat_cloud_summary_flag). For the N2O and CH4 data we apply a more stringent cloud filter and further inspect data where the EUMETSAT L2 PPF cloudiness assessment summary flag indicates a possibility of small cloud contamination. For respective data we require that the EUMETSAT L2 processing cannot clearly attribute a value for fractional cloud cover, which means that the cloud signals are rather weak (see Sect. ). We use this more stringent cloud filtering for N2O and CH4 because both species have relatively weak atmospheric variabilities that are very similar to the errors estimated for a small cloud coverage (10 % coverage with opaque cumulus clouds or 25 % coverage with cirrus clouds).

Furthermore, we filter according to the retrieval fit noise and estimated atmospheric temperature errors. The respective errors are provided for each observation in the MUSICA IASI standard file output variables with the suffix _error. For HNO3 we calculate the retrieval fit noise and the estimated temperature errors for the 10–35 km partial column-averaged mixing ratios according to Eq. (), whereby we reconstruct the noise covariance matrix for HNO3 according to Eq. () and generate the atmospheric temperature covariance according to Eq. () using the MUSICA IASI standard file output variables musica_at_apriori_amp and musica_apriori_cl for setting up the values of vamp,i and σcl,i, respectively.

In order to ensure that the time series signals or horizontal patterns are not significantly affected by varying sensitivity and vertical resolution, we filter H2O, δD, N2O, and CH4 data according to the averaging kernel metrics MR, LWpD, and C-Alt. This filters out data with anomalous vertical sensitivities. For HNO3 we calculate the 10–35 km partial column-averaged mixing ratio averaging kernels according to Eq. () and filter for good sensitivity by requiring a diagonal entry close to unity.

The filter threshold values of LWpD and C-Alt​​​​​​​ are defined relative to the a priori assumed vertical correlation length (provided in the variable musica_apriori_cl). We use threshold values for these ratios that are constant for all altitudes, which allows for increased LWpD and C-Alt values in the case of an increased vertical correlation length (for altitudes with a larger correlation length, higher values of LWpD andC-Alt can be accepted).

In this section we give an example of the temporal continuity of the data. Figure  depicts a time series at the mid-latitudinal site of Karlsruhe, Germany, between October 2014 and June 2021 of MUSICA IASI trace gas retrieval products. For all trace gases, except for δD, we have good temporal coverage with no significant data gaps caused by the comprehensive data filtering. Concerning δD, there is a reduced data volume in winter mainly due to filtering out data with reduced sensitivity (measurement response below 0.8). It is worth noting that for the {H2O, δD} pair optimal estimation product – generated a posteriori according to – we achieve a significantly better measurement response.

We observe typical seasonal cycles for all species. The seasonal cycles of H2O and δD follow the seasonal cycle of temperature. In winter H2O concentrations can be as low as 100 ppm and δD values can be below -350 ‰. In summer the maximum values are about 8000 ppm and -150 ‰. The concentrations of N2O and CH4 at 10.9 km a.s.l. are lowest in winter–spring and highest in summer–autumn. This cycle is linked to the vertical shift of the tropopause altitude: in winter–spring the 10.9 km altitude is much more strongly affected by the stratosphere (where N2O and CH4 are decreasing with height) than in summer–autumn. Concerning HNO3 we observe the highest values in winter–spring, which might indicate the detection of air masses with an Arctic stratospheric history (Arctic winter stratospheric HNO3 mixing ratios are particularly large).

In this section we give an example of the good daily global coverage achieved by high-quality MUSICA IASI products. Figure  depicts the data retained during 24 h when using the filter setting listed in Table . For our example we choose 1 February and 1 August 2018 and plot the data for the same altitudes as in Fig. . For all data products, except for δD, we have very dense global coverage. Areas with missing data are mostly linked to the cloud filtering. The reduced data coverage for δD in the middle and high latitudinal winter hemispheres is due to δD measurement response values lying below 0.8 (we achieve a significantly better measurement response and thus horizontal coverage for the optimal estimation {H2O,δD} pair product generated according to ).

The highest H2O concentrations at 4.2 km are observed at low latitudes where temperatures are generally highest. However, there are also low latitudinal areas with rather low H2O concentrations, for instance in the eastern Pacific on 1 August 2018, which indicates a region where large-scale subsidence is prevailing. The δD values at 4.2 km are also highest at low latitudes but with a stronger zonal variability. For high tropical H2O concentrations, δD values can be relatively high (for instance on 1 February 2018 in the tropical Atlantic) or relatively low (for instance on 1 February 2018 in the tropical Indian Ocean). This indicates that δD data contain information that is complementary to the H2O data.

For N2O and CH4 at 10.9 km we observe maximum concentrations at low latitudes and rather low values in the polar regions. The reason for this is that at high latitudes the 10.9 km altitude is strongly influenced by low stratospheric concentrations, whereas in the tropics the 10.9 km altitude is representative of the upper troposphere, where concentrations are higher. This means that the concentrations observed at 10.9 km reflect to a large extent the altitude of the tropopause. On 1 August 2018 we observe for both trace gases a clear gradient between the Northern Hemisphere and Southern Hemisphere, whereas there is no significant gradient on 1 February 2018. This is caused by higher tropospheric concentrations of both trace gases in the Northern Hemisphere. On 1 August 2018 the stratosphere affects the 10.9 km altitude more strongly in the Southern Hemisphere than in the Northern Hemisphere and we observe particularly strong gradients. On 1 February 2018 it is the other way round and the tropospheric concentration gradients are counterbalanced by the tropopause altitude effect.

The global maps of the HNO3 10–35 km partial column-averaged mixing ratios show very low values in the tropics and the highest values in polar regions. However, in the Antarctic low values are also found in winter because at very low temperatures (<195 K) polar stratospheric clouds (PSCs) are formed on which HNO3 condensates. In the Arctic winter temperatures are generally not that low and PSCs and consequently low HNO3 values are mainly restricted to areas with a local mountain lee wave occurrence.