Atmospheric Radiation Measurement (ARM) airborne field campaign data products between 2013 and 2018

Airborne measurements are pivotal for providing detailed, spatiotemporally resolved information about atmospheric parameters and aerosol and cloud properties, thereby enhancing our understanding of dynamic atmospheric processes. For 30 years, the US Department of Energy (DOE) Office of Science supported an instrumented Gulfstream 1 (G-1) aircraft for atmospheric field campaigns. Data from the final decade of G-1 operations were archived by the Atmospheric Radiation Measurement (ARM) Data Center and made publicly available at no cost to all registered users. To ensure a consistent data format and to improve the accessibility of the ARM airborne data, an integrated dataset was recently developed covering the final 6 years of G-1 operations (2013 to 2018, https://doi.org/10.5439/1999133; Mei and Gaustad, 2024). The integrated dataset includes data collected from 236 flights (766.4 h), which covered the Arctic, the US Southern Great Plains (SGP), the US West Coast, the eastern North Atlantic (ENA), the Amazon Basin in Brazil, and the Sierras de Córdoba range in Argentina. These comprehensive data streams provide much-needed insight into spatiotemporal variability in the thermodynamic quantities and aerosol and cloud properties for addressing essential science questions in Earth system process studies. This paper describes the DOE ARM merged G-1 datasets, including information on the acquisition, data collection challenges and future potentials, and quality control processes. It further illustrates the usage of this merged dataset to evaluate the Energy Exascale Earth System Model (E3SM) with the Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package.

1 Introduction

The Earth's climate is changing due to human activities such as fossil fuel burning and deforestation. Atmospheric research is critical for understanding the causes and effects of climate change and for developing strategies to mitigate its impacts (IPCC, 2021). Airborne measurements provide a unique observational perspective within the broad objective of understanding atmospheric processes relevant to climate change. Aircraft are often the only platform from which in situ observations of chemical and physical parameters through the depth of the troposphere can be obtained. These platforms allow one to follow chemical and physical processes through the use of pseudo-Lagrangian flight patterns, their spatial coverage is usually greater than that obtained from surface sites, and their observations have a longer time duration than can be obtained from a (non-geostationary) satellite overpass. Moreover, vertical profiles can be measured, allowing researchers to obtain vertical profiles of atmospheric states, aerosols, clouds, and trace gas parameters with an accuracy that is not attainable with remote sensing. Such parameters are essential for understanding atmospheric physical and chemical processes. Compared with ground observatories, airborne measurements can provide greater spatiotemporal characterization in a limited time, contributing to a critical, otherwise missing context. This capability is particularly useful for studying rapidly evolving atmospheric phenomena like wildfires, hurricanes, and dust storms. Hence, airborne measurements are a valuable tool in atmospheric studies, as they offer several advantages over ground- or satellite-based measurements (Wendisch et al., 2013; Schumann et al., 2013; Petzold et al., 2013; McQuaid et al., 2013; McFarquhar et al., 2011; Krämer et al., 2013; Brenguier et al., 2013).

The US Department of Energy's Atmospheric Radiation Measurement (ARM) user facility (Mather and Voyles, 2013) has provided long-term measurements of atmospheric properties by operating ground-based observatories (fixed and mobile) as well as aerial facilities. Established in 2006, the ARM Aerial Facility (AAF) has led 14 field campaigns using state-of-the-art instruments aboard numerous aircraft (Schmid et al., 2014). Initially, the AAF data were archived individually for each instrument in the International Consortium for Atmospheric Research on Transport and Transformation (ICARTT) file format (Thornhill et al., 2011), which was developed to fulfill the data management need in 2004. A major strength of the ICARTT file format is its easy-to-use and standard approach to sharing airborne datasets to facilitate broad collaborative scientific research among airborne observation, atmospheric modeling, and satellite observation communities. However, shortcomings of the ICARTT file format have also long been identified: it is (1) not as efficient as binary formats for data collection and storage and (2) not suitable for extensive multidimensional data (https://www.earthdata.nasa.gov/esdis/esco/standards-and-practices/icartt-file-format, last access: 24 April 2023). ARM has started efforts to convert the historical field campaign data into Network Common Data Form (NetCDF) format (McCord and Voyles, 2016; Rew et al., 1997) – a widely used self-describing data format that supports creating, accessing and sharing array-oriented scientific data.

Additionally, researchers commonly use data analysis software (e.g., Python) that includes functions and library packages that make working with NetCDF files easier than text-like files. Using the NetCDF format, various airborne measurements can be easily combined to generate a merged dataset, thereby relieving end users of the burden of combining data from different data sources. The merged dataset can also aid the research community that uses the abundant ARM aerial data obtained from different field campaigns for diverse science objectives, as detailed in Table 1.

The demand for airborne observations continues to increase with the increase in weather and climate model complexity, as well as the increasing interest in small-scale physical and chemical processes. Such observations are needed to assess and validate process-level understanding. To provide a comprehensive view of atmospheric properties, it is desirable to integrate different types of data into a single file, thereby providing efficient data access for researchers to study the interactions among aerosols, clouds, and trace gases under various atmospheric conditions in order to understand effects on atmospheric processes and climate. Having all data in one file simplifies data management and reduces the number of files that need to be stored, shared, and accessed. It also minimizes the chances of data loss or errors during file transfers. In a single file, all measured variables are mapped into the same timestamp (e.g., 1 Hz for this study). More details are discussed in Sect. 4. These “merge files” are developed to assist researchers in performing more complex analyses, such as studying the relationships between different types of atmospheric data and carrying out more comprehensive studies with larger datasets. In addition, the merged data will hopefully encourage experimentalists and modelers to collaborate, thereby combining their expertise and resources to obtain a more complete understanding of the atmospheric phenomena. Thus, after standardizing the AAF data into a NetCDF file format for each field campaign, we used the ARM Data Integrator (ADI, https://github.com/ARM-DOE/ADI, last access: 24 April 2023) to retrieve and prepare data from each measurement and integrate them into a merged dataset (Gaustad et al., 2014).

This study provides an overview of the airborne datasets collected during seven field campaigns (listed in Table 1) between 2013 and 2018, an introduction to the integrated datasets, and a guide for users to access these datasets on the ARM data archive. Although airborne field campaigns can lead to highly significant scientific findings, they often have a lengthy timeline for generating research papers. Meanwhile, due to pressure to publish quickly and the limited length of funding cycles, researchers do not have enough time to fully leverage field campaign data. In this study, one of the objectives is to draw researchers' attention to these valuable field data, encouraging them to revisit underutilized datasets to reveal new insights. In Sect. 2, this paper provides an overview of the objectives, flight information, and measurements of the seven field campaigns that the AAF carried out with the G-1 aircraft between 2013 and 2018. Section 3 has three subsections: data quality is discussed by comparing the in situ measurements with other measurements; we briefly outline how to use the merged dataset to evaluate aerosol–cloud interactions represented in Earth system models; and we outline data collection challenges, lessons learned, and future potentials of these airborne data. Further detail on the data structure is given in Sect. 4. In addition, a summary section describes the potential of this merged dataset after explaining the data file structure and availability of the data.

2 Campaign objectives and flight patterns

2.1 AAF-supported field campaigns between 2013 and 2018

The ARM program is a US Department of Energy scientific user facility that aims to provide observation data to improve our understanding of the Earth's atmosphere and its interactions with the land surface and oceans. One of the key components of the ARM program is to conduct periodic field campaigns, which are intensive measurement periods at specific locations focused on specific scientific questions. These field campaigns involve deploying state-of-the-art instruments with both ground station and airborne platforms to collect measurements of atmospheric states, radiation, clouds, precipitation, aerosols, and trace gas variables, often including collaborations with other research programs and institutions. Table 1 shows the campaign locations, flight hours, and scientific objectives of each field study carried out by the AAF between 2013 and 2018. The “word cloud” depicted in Fig. S1 encapsulates the content of all AAF-supported field campaign publications, showing that the AAF has played a pivotal role in advancing research across various important atmospheric domains. The visual representation highlights a concentration of terms related to key atmospheric topics, including aerosols, clouds, precipitation properties, and the intricate processes governing their interactions. This observation underscores the multifaceted support provided by the AAF to the scientific community, contributing to the exploration and understanding of crucial atmospheric phenomena.

Table 1 A brief summary of the seven AAF field campaigns between 2013 and 2018.

* ARM site identifier was included with each field campaign, such as OSC for the BBOP campaign.

2.2 Description of the airborne platform and sensors

The ARM Aerial Facility (AAF) took over operations of the G-1 research aircraft in 2010. The G-1 is a medium-sized business aircraft that provided atmospheric measurements for various DOE programs from 1989 to 2018 (Fig. 2). The aircraft is powered by two turboprop engines, and it has a 1900 kg cabin payload and an operational range of approximately 4000 km for ferry flight at high altitudes and 800 km for boundary layer sampling, making it well-suited for deployments to remote locations. The G-1 can continuously fly for 3–4 h with a full payload (5 h with minimum payload) at a cruising speed of approximately 100 m s⁻¹ up to a maximum altitude of approximately 7.6 km, which allows it to access a wide range of atmospheric conditions and altitudes for measurement purposes.

Figure 1 Flight tracks overlaid with the campaign location map from seven AAF field campaigns between 2013 and 2018 using MATLAB^®.

Figure 2 The G-1 research aircraft flying over the ARM eastern North Atlantic (ENA) site (image courtesy of Ed Luke, Brookhaven National Laboratory).

The ARM G-1 is equipped with a suite of instruments and sensors for making measurements of atmospheric parameters (including temperature, humidity, pressure, winds, and turbulence); radiation; concentrations of atmospheric gases; and types, concentrations, and sizes of aerosol, cloud, and precipitation particles (as shown in Table 2). These instruments and sensors are mounted on the aircraft's exterior and inside the cabin. The measurement variables for individual data streams are listed in Table S1 and are used as input data for the merged data product. Note that some measurements, including those related to redundant cloud measurements and radiation, are not currently incorporated into the merged dataset described here due to the complexity of the existing data. However, these measurements remain under consideration for future inclusion, reflecting a commitment to expanding the dataset's scope as technology and analytical capabilities evolve. Additionally, photos, videos, and some guest instruments are nontraditional data types included in the ARM individual archive but not with these merged data. Driven by the various campaign objectives and payload limitations, the measurements available for each field campaign are listed in Table S2. The data collected by these instruments are used to improve our understanding of the Earth's climate system and to support the development of climate models and other tools for studying and predicting atmospheric and climate phenomena (Mei et al., 2020).

Table 2 The ARM G-1 payload instruments that were included in the merged data product.

3 Data quality evaluation – consistency among observations

The AAF's airborne measurements are considered accurate and reliable because they are obtained directly from the atmosphere using well-calibrated instruments. Numerous prior studies have systematically assessed ARM data quality, employing methods such as laboratory evaluation based on community-accepted standards, comparing similar properties across different instruments, and conducting intercomparisons across diverse platforms (from ground to airborne or airborne to airborne) (Bond et al., 1999; Lance et al., 2010; Kassianov et al., 2015, 2018; Mei et al., 2020; Zawadowicz et al., 2021; Kulkarni et al., 2023; Tang et al., 2023; Zhang et al., 2023). Table S3 lists AAF measurements and uncertainties in the atmospheric properties, including temperature, humidity, aerosol concentrations, cloud particle sizes, and radiation levels (Mei et al., 2020).

The AAF airborne data are also often used as a benchmark or standard for other measurements, especially those from remote-sensing technologies such as satellites, ground-based radars, and lidars. Junghenn Noyes et al. (2020) validated remote-sensing retrievals with help from ground-based and airborne measurements. Their study enhanced the understanding of smoke particle behavior and its implications for remote sensing (Junghenn Noyes et al., 2020). Mech et al. (2020) showcased how integrating airborne data into the PAMTRA (Passive and Active Microwave TRAnsfer) validation process enhances the model's skill with respect to accurately simulating microwave measurements. The detailed comparison between simulated and observed data helps understand the model's performance under real-world conditions, leading to a more robust and reliable tool for atmospheric research (Mech et al., 2020). Yang et al. (2019) developed a new method to estimate supersaturation fluctuations in stratocumulus clouds using ground-based remote-sensing retrievals. Then, they used the airborne data to validate these estimations (Yang et al., 2019). Wu et al. (2020) retrieved profiles of marine boundary layer (MBL) cloud and drizzle microphysical properties from ground-based observations, validated by aircraft measurements over the Azores (Wu et al., 2020). Zhang et al. (2023) evaluated cloud droplet number concentrations using multiple ground-based methods validated through aircraft in situ measurements (Zhang et al., 2023).

Furthermore, research based on collected aircraft data led to advancements in the characterization and understanding of atmospheric processes (Martin et al., 2017; Fast et al., 2011, 2019a, b; Varble et al., 2021; Wang et al., 2022). For example, various studies have utilized aircraft measurements to characterize aerosol and cloud properties while also advancing the understanding of aerosol chemistry and cloud microphysical properties and processes, including investigations over the North Atlantic, Amazon Basin, and Southern Great Plains (Shrivastava et al., 2019; Shilling et al., 2018; Zawadowicz et al., 2021; Wang et al., 2023a; Fast et al., 2024) The airborne data have also been used to examine the vertical variability in the aerosol properties over the Southern Great Plains, contributing to a better understanding of the distribution and impact of aerosols at different atmospheric levels (Wang et al., 2016; Fast et al., 2022).

This study further demonstrates the comparison of AAF data with ground-based remote-sensing retrievals. The ARM G-1 aircraft was deployed above or near the eastern North Atlantic (ENA) and Southern Great Plains (SGP) sites during field campaigns like ACE-ENA and HI-SCALE. Various instruments employing diverse observational techniques measured atmospheric parameters, aerosol, and cloud properties from the ground and airborne perspectives. These coordinated deployments enable a thorough assessment of robustness and statistical representativeness across co-located measurements.

Comparing airborne and ground-based measurements involves evaluating data from two platforms that differ with respect to their spatial and temporal resolutions and measurement techniques. Thus, three potential biases exist in the measurements – spatial, temporal, and instrumental. The instrumental bias is typically due to the differences in sensors, calibration, and data-processing techniques between the two platforms. Airborne measurements usually provide in situ spatiotemporal data over leveled flight legs at different altitudes and can capture data over various, even difficult-to-access, terrains. Meanwhile, ground-based remote-sensing data usually provide continuous monitoring at a fixed location with limited spatial coverage or less vertical resolution. Our efforts focus on minimizing the temporal and spatial biases to ensure accurate and meaningful comparisons. We selected the comparison period by aligning the data acquisition times for both airborne and ground-based measurements as closely as possible. To ensure that both airborne and ground-based measurements are georeferenced accurately. For instance, ground-based remote sensing uses height or the altitude above ground level (a.g.l.) as the vertical geographic coordinate. In contrast, airborne data usually use the mean sea level (m.s.l.) altitude, which can be converted to the altitude above ground level. We then use interpolation techniques to match the spatial resolutions of airborne and ground-based data.

3.1 Airborne data quality control

3.1.1 Aircraft-integrated meteorological measurement system data

The aircraft platform velocity, position, and attitude were monitored by four redundant sensors: DSM, C-MIGITS, VectorNav, and AIMMS-20, as shown in Table 2. The DSM is the primary choice with respect to the data source for the merged dataset, followed by the C-MIGITS, AIMMS-20, and VectorNav. For static temperature measurement, the uncertainty in the field data is ±0.5 K. The static pressure has a measurement uncertainty of 0.5 hPa. The standard measurement uncertainties were ±2 K for the chilled-mirror hygrometer. The AIMMS-20 provides both meteorological measurements and wind vectors (https://aventech.com/products/aimms20.html, last access: 24 April 2023). The relative humidity sensor has an accuracy of 2 % in the operating range of 0 %–100 %. A calibration flight pattern was performed with each installation of the AIMMS-20 in each field campaign, improving the high-resolution wind data accuracy. The AIMMS-20 calibration uses two different procedures: first, an aerodynamic calibration maneuver helps to determine aerodynamic errors induced by the aircraft itself; then, the aircraft is operated to complete an inertial system calibration maneuver and capture minor alignment errors (i.e., cross-axis error) between the gyros, accelerometers, GPS antenna baseline, and the primary reference frame of the inertial measurement unit (IMU). Combining the measurements from the instruments that make up the AIMMS-20 (the air data probe – ADP; the GPS; and the IMU) provides the wind speed with an accuracy of 0.5 m s⁻¹ for the north and east components and an accuracy of 0.75 m s⁻¹ for the vertical wind component.

The AIMMS-20 wind parameters, such as wind speed and wind direction, were compared with the ground-based wind parameters retrieved using the ARM Doppler lidar (DL) (Newsom et al., 2019), as shown in Fig. 3. The ARM DL is an active remote-sensing instrument that provides temporally and range-resolved measurements (Newsom and Krishnamurthy, 2022). The DOE ARM user facility operates several scanning coherent Doppler lidar systems in the near-infrared (1.5 µm) at ARM's ground-based observatories and mobile facilities (Newsom et al., 2017). The Doppler lidar horizontal wind profiles value-added product (DLPROF-WIND VAP, https://www.arm.gov/capabilities/vaps/dlprof-wind, last access: 24 April 2023) provides accurate height-resolved measurements of wind speed and direction at a 15 min resolution (Newsom et al., 2019). Figure 3 compares G-1 aircraft winds to DL winds for level flight legs under cloudless conditions during the ACE-ENA field campaign. The aircraft data were averaged to match the DL time interval when the aircraft location was within a 3 km distance of the ARM ground site. Limited by frequent clouds and the DL data availability, only leveled flight legs from 12 flights between 26 June and 19 July 2017 were included in the comparison. We achieved a reasonably good comparison between the AIMMS-20 and DL wind parameters, especially for the wind direction. The wind speed plot exhibits increased dispersion, particularly at lower wind speeds (<5 m s⁻¹), possibly due to the heterogeneous nature of wind speeds at different heights in the Doppler lidar data. Notably, data from a single aircraft flight path may not accurately reflect the values measured by the Doppler lidar. It should be noted that there are fundamental differences in aircraft-based wind measurements (in situ, spatial averages) and the Doppler lidar technique (single point, time/height averaged), which never allows perfect agreement between the two datasets.

Figure 3 Wind parameter comparison between the AIMMS-20 and Doppler lidar, with the color scale showing aircraft altitude, for cloudless flight legs during the ACE-ENA campaign.

3.1.2 Aerosol payload data

Aerosol measurements aboard the G-1 platform were performed downstream of an isokinetic inlet with a Nafion dryer to ensure a relative humidity below 40 % for the air sample. The isokinetic aerosol inlet was designed and built by Brechtel Manufacturing Inc. (Hayward, CA) and modified by the Pacific Northwest National Laboratory (PNNL). Previous closure studies have shown that the isokinetic inlet allowed sampling of the aerosols up to a 5 µm aerodynamic diameter, and the transmission efficiency is around 50 % at 1.5 µm (Kassianov et al., 2015, 2018; Mei et al., 2020; Kassianov et al., 2021). The best-estimate aerosol size distribution (BEASD) data product was created by merging the aerosol size distribution from several (up to four) aerosol and cloud sensors under dry conditions (<40 % RH). Two aerosol spectrometers were used as primary data sources: the fast integrated mobility spectrometer (FIMS; 10–600 nm size range; 30 log-spaced bins, 16.31 bins per decade) and the passive cavity aerosol spectrometer (PCASP, 0.095–2.9 µm size range). As shown in Fig. 4, we demonstrate a reasonable agreement between the integrated total number concentration from the BEASD and the total number concentration measured by the condensation particle counter (CPC, TSI, model 3772, >10 nm). This CPC was calibrated following the World Calibration Center for Aerosol Physics (WCCAP) guidelines. Compared with the laboratory standard CPC, the typical uncertainty is ∼10 % (Mei et al., 2020). Two cloud probes were used as a secondary source of aerosol measurements to cover the super-micron size range: the cloud aerosol spectrometer (CAS, 0.55–12.73 µm) and the fast cloud droplet probe (FCDP, 0.75–13.49 µm). Aerosol chemical composition, measured by the Aerodyne high-resolution time-of-flight aerosol mass spectrometer (HR-ToF-AMS), was used to estimate the aerosol refractive index (RI), which is necessary for the correction of the aerosol equivalent optical size into the equivalent geometric size (Li et al., 2023; Hand and Kreidenweis, 2002; Freedman et al., 2009).

Figure 4 Aerosol total number concentration comparison between the G-1 CPC 3772 and the best-estimate aerosol size distribution during the ACE-ENA flights.

The scattering and absorption coefficients are two essential parameters for understanding the optical properties of atmospheric aerosols, as they describe the scattering and attenuation of sunlight and the visibility of the atmosphere. The payload of the G-1 incorporated two aerosol optical instruments to measure these properties. A nephelometer (TSI, model 3563) was used to measure light scattering by aerosol particles at three wavelengths (Uin and Goldberger, 2020). It uses a light source to illuminate a sample of aerosol particles and measure the intensity of the total and backscattered light. The scattering coefficient from a nephelometer is defined as the ratio of the scattered light flux to the incident light flux and is typically expressed in units of inverse megameters (Mm⁻¹). A particle soot absorption photometer (PSAP, Radiance Research) is used to estimate aerosol absorption by measuring the attenuation of a light beam passing through aerosols deposited on a filter (Springston, 2018). After correcting for the filter effect and scattering impacts on the absorption values, the amount of light absorbed by the particles for three wavelengths was recorded (Bond et al., 1999). Under dry conditions, the accuracies for the scattering and absorption coefficients are 25 % and 20 %, respectively (Rosati et al., 2016). The uncertainties in those airborne measurements might be even larger due to the complex field conditions. The sum of the absorption and scattering of aerosol particles (i.e., aerosol extinction) provides a measure of the effect of aerosols on radiant energy that passes through the atmosphere.

In addition, the aerosol extinction coefficient can be calculated from the aerosol size distribution and chemical composition using Mie theory with certain assumptions (Mie, 1908; Bohren and Huffman, 1998). Here, assuming spherical particles and a homogeneous composition in the estimated size range, we estimated the extinction coefficients of the aerosol particles and compared the values with the in situ measurements from the summation of the values from the PSAP and nephelometer in Fig. 5. The comparison data were only from level flight legs when aerosol sampling was under dry and isokinetic conditions. The BEASD estimation and the nephelometer and PSAP summation shown on the left-hand plot in Fig. 5 achieved a reasonably good agreement. The dots on the scatter plot shown on the right-hand side of Fig. 5 are color-coded by the relative humidity. This scatter plot indicates that the simple assumption, which ignores how the aerosol properties vary with particle size, shape, and refractive index and also neglects the ambient relative humidity effect, introduces uncertainty in the aerosol properties.

Figure 5 Aerosol extinction coefficient comparison between the estimated values based on the BEASD size distribution and the summation of the in situ measurements from the nephelometer and PSAP during the ACE-ENA flight for the leveled flight legs.

3.1.3 Cloud payload data

A variety of cloud optical sensors were deployed during the seven AAF campaigns. The cloud measurement methods can be separated into light scattering (FCDP) and shadow imaging (2D-S and HVPS), covering a wide range from small cloud droplets to precipitation elements. The FCDP measures cloud droplets in the size (diameter) range of 2–50 µm. Droplets are detected and sized depending on how much light they scatter in a specific angular range when illuminated by a focused laser beam (Lance et al., 2010). The 2D-S and HVPS are optical array probes that restore images from passing hydrometeors' shadows. Using two cameras positioned at different angles to create a stereo image of each cloud particle, 2D-S and HVPS provide information about the particle's size, shape, and orientation at different size ranges (Glienke and Mei, 2020, 2019). Based on previous studies, the undercounting bias of measured droplets between 3 and 20 µm diameter is around 20 %, whereas the uncertainty is up to 50 % for droplets with a diameter larger than 20 µm (Glienke et al., 2023; Mei et al., 2020). In addition, a merged size distribution based on the FCDP, 2D-S, and HVPS was created to cover the size range of cloud elements from 2 to 9075 µm.

Figure 6 Cloud droplet number concentration comparison between the integrated value from the merged cloud droplet size distribution and the ground-based Raman lidar retrieval value on 18 July 2017, during the ACE-ENA field campaign.

Figure 6a demonstrates one cloudy-day (stratocumulus cloud) case and shows three lines representing the cloud-top height, cloud-base height (retrieved from micropulse lidar), and aircraft G-1 altitude for the 18 July 2017 flight. The cloud extends from about 2 to 5 km altitude, with the cloud-top and cloud-base heights varying over time. The G-1 flew level through different cloud layers (09:40–10:35 and 11:00–11:40 UTC). The G-1 also porpoises in and out of the cloud between 10:27 and 10:33 UTC.

Figure 6b compares the time series of the total droplet concentration from three different sources. The blue stars depict the droplet concentration retrieved from the Raman lidar measurements, the brownish dots show the droplet concentration measured by the G-1 cloud probe, and the black stars show the droplet concentration by the G-1 measurements averaged to the lidar retrieval times. The cloud droplet number concentration (N_{drop, retrieval}) is derived using the algorithms/methods developed by Snider et al. (2017), based on the ground-based Raman lidar particulate extinction profile from the Raman Lidar Vertical Profiles – Feature Detection and Extinction (RLPROF-FEX) value-added product (Chand et al., 2023). The N_{drop, retrieval} achieves good results for stratocumulus cloud, while the cloud liquid water content profiles are closer to adiabatic/pseudo-adiabatic conditions. The lidar retrievals and the G-1 cloud probe measurements agree well, while the lidar-retrieved value has a lower droplet concentration near the cloud base between 9:40 and 9:55 UTC and between 11:10 and 11:40 UTC.

A statistical comparison of the total droplet number concentration (N_drop) between the G-1 measurements and the lidar retrieval shows that the two measurements are highly correlated in the concentration ranges up to 200 cm⁻³, and the G-1 cloud probe measures a slightly higher droplet concentration than the lidar retrieval. Recent studies (Tang et al., 2022, 2023; Zhang et al., 2023) have also pointed out the need to develop appropriate criteria to quantify the cloud remote-sensing retrievals better using in situ measurement, as the lidar is looking straight up and measuring clouds passing over, while the aircraft is flying legs overhead and probing different cloud layers. The large discrepancy might be due to the cloud retrievals mainly capturing the cloud base, whereas the G-1 samples through a portion of the cloud.

3.2 Earth system model aerosol–cloud diagnostics package evaluation

An Earth system model (ESM) aerosol–cloud diagnostics package, known as ESMAC Diags (Tang et al., 2022, 2023), has been developed to facilitate the routine evaluation of aerosol, cloud, and aerosol–cloud interactions quantities simulated by the DOE's Energy Exascale Earth System Model (E3SM) using in situ surface, aircraft, and ship measurements. ESMAC Diags reads in datasets from selected field campaigns and model outputs with some processing and quality controls; it then generates a set of diagnostics plots and metrics, such as the mean, root-mean-square error, and correlation of aerosol and cloud variables. To run the simulation for comparison with aircraft field campaign data, we configure the model according to the Atmospheric Model Intercomparison Project protocol (Gates et al., 1999), using real-world initial conditions and nudging simulated winds toward Modern-Era Retrospective analysis for Research and Applications (MERRA-2) reanalysis data (Gelaro et al., 2017). Then, we save the hourly output for the field campaign area and utilize an “aircraft simulator” strategy (Fast et al., 2011) to extract the closest model grid and level-matching aircraft measurements. More details about the model configurations can be found in Tang et al. (2022).

With this diagnostic package, various types of diagnostics and evaluation metrics are performed for the aerosol number, size distribution, chemical composition, cloud condensation nuclei (CCN) concentration, and various meteorological quantities to assess how well the E3SM represents observed aerosol properties across spatial scales (Tang et al., 2022). Data from two ARM airborne field campaigns (HI-SCALE and ACE-ENA) have been included in the current version of the ESMAC Diags package. The integrated dataset discussed in this study provides a more consistent input data format than prior data files for ESMAC Diags. One example is shown in Fig. 7. During the intensive operation period (IOP) in the ACE-ENA field campaign (between 15 June and 20 July 2017), three aerosol number concentrations (for aerosol particles larger than 3, 10, and 100 nm) and the cloud condensation nuclei concentration (at 0.1 % supersaturation) were compared using an E3SM model (version 2) simulation with the ESMAC Diags package. Although the E3SM qualitatively reproduced the observation, it overestimated accumulation-mode aerosols and CCN concentration over the ENA regions. In addition, larger discrepancies were observed at lower altitudes (<500 m a.g.l.) for the aerosol number concentrations (>3 and >10 nm), which might be due to the weak representation of nucleation-mode aerosol in the model. It did not capture the vertical variation in the CCN concentration, which indicates that process-level improvement is still needed for the E3SM over the Atlantic Ocean.

Figure 7 An example of the E3SM evaluation using an integrated flight dataset with the ESMAC Diags package (15 June to 20 July 2017, ACE-ENA field campaign). The box represents the 25th and 75th percentiles, while the bars represents the 5th and 95th percentiles.

3.3 Data collection – challenges and future potential

We summarize the challenges and data collection limitations encountered during the seven field campaigns in Table 3. The lessons learned from these campaigns suggest that ensuring data quality and enhancing data collection variability require the following key strategies:

• regular sensor calibration to maintain accuracy with scheduled, impromptu, and well-documented validation methods ensuring reliable measurements;

• cross-validation and monitoring of sensor performance using redundancy, dataset fusion, and statistical techniques to identify inconsistencies and malfunctions;

• diversified sampling strategies to ensure comprehensive data representation across varying conditions;

• the leveraging of model simulations and statistical studies based on previous measurements to refine data collection methods and anticipate potential issues, leading to more effective and targeted data acquisition.

These combined approaches ensure robust data collection, improve measurement accuracy, and provide reliable data products for the community of users.

Future work can expand data collection and merge and facilitate further investigations into atmospheric chemistry, aerosol properties, aerosol–cloud interactions and their representation in Earth system models. To support future research needs, the ARM Data Center plans to work with the AAF instrument mentors and community experts to standardize this merged data product. Future deployments will use a new airborne platform (Challenger 850) and include more baseline airborne measurements (https://arm.gov/capabilities/instruments?type[0]=armobs&category[0]=Airborne Observations, last access: 24 April 2023). For example, we plan to add the liquid water content measurements from the Multi-Element Water Content System (https://arm.gov/capabilities/instruments/wcm-air, last access: 24 April 2023) and solar radiation measurements from multi-filter radiometers (https://arm.gov/capabilities/instruments/mfr-air, last access: 24 April 2023) into the future merged dataset.

In addition to the baseline measurements, we plan to offer data integration options through the ARM Data Integrator tool (details in Sect. 4), allowing campaign principal investigators or community users to flexibly incorporate additional data into the merged data product. This approach provides users with the ability to target specific science themes. For example, one proposed data product is to include additional aerosol optical properties with this AAF merged dataset. The atmospheric community can expand research on the evolution of aerosol particles from wildfires, particularly on how different combustion phases (flaming vs. smoldering) result in varying chemical compositions and quantities of emitted aerosols with additional chemical composition and gas-phase concentration data. This custom-built merged dataset is crucial for improving air quality models and understanding the climate impacts of biomass burning. Continuing to investigate the influence of urban pollution on natural aerosol formation, similar to the studies conducted in the Amazon during GoAmazon2014/5, will provide new insights into aerosol composition. Potential data for such a study could be non-airborne remote-sensing data. Combining airborne data with ground-based remote-sensing data allows the exploration of interactions between different aerosol types and evolving cloud and precipitation patterns. Quantifying these interactions can improve models and understanding, aiding the development of strategies for mitigating anthropogenic impacts on natural environments.

By strategically combining long-term ground-based remote-sensing measurements with high-resolution airborne data, researchers can achieve more robust analyses of atmospheric processes, leading to more accurate scientific findings and better-constrained models. For instance, ground-based sensors can continuously monitor a specific location, while targeted airborne missions can capture critical in situ measurements during specific events that are not retrievable by remote sensing to better study cloud evolution, pollutant transport, or extreme weather.

Table 3 Challenges and data collection limitations of the seven AAF field campaigns between 2013 and 2018.

Table 4 The naming convention of the merged data product.

4 Code and data availability

The ARM data management system treats all data from the G-1 airborne deployments as field campaign data streams, meaning the data were collected during the intensive observational periods rather than over long-term observation. The data (https://doi.org/10.5439/1999133; Mei and Gaustad, 2024) in this paper were produced following ARM data file standards and archived through the ARM data ingestion process (Prakash et al., 2016) under a Creative Commons license. In 2016, the ARM G-1 raw instrument data were directly “ingested” into NetCDF format and archived automatically during and after flight operations. To provide a dataset with a uniform data format, the G-1 payload scientists (ARM mentors) reprocessed data for field campaigns between 2013 and 2015 and converted the historical data from ICARTT to NetCDF. Note that a specific directory was created for reviewers to access the data at https://doi.org/10.5439/1999133 (Mei and Gaustad, 2024).

The ARM data system uses a multitiered data-processing approach (Prakash et al., 2016) that iteratively processes the instrument data to produce higher-level data products. Data are first processed from the instrument's raw data format (data level 00) to NetCDF format. During this initial processing, the conversion of geophysical units and the application of calibration factors are performed as appropriate (documented in a- and b-level data products). Quality controls can also be applied (creating b1-level data files). Additional processing can be added to further increase the level of these files with “higher value” to store as a data level “c1”. For example, mentor-edited data files with additional quality improvement calibration are usually considered the c-level data product.

The content of ARM data files is structured in three main sections: dimensions, variables, and global attributes. As time-series measurements, the time dimension of ARM products is considered “unlimited”. Per NetCDF-3 requirements, the unlimited dimension “time” is the first dimension of a variable that uses the time dimension. The variables encompass coordinate variables (reporting dimension values), primary measurements (recommended for scientific use), supporting measurements (e.g., diagnostics and quality), and location variables (detailing latitude, longitude, and altitude). Variables are equipped with supporting attributes to facilitate the user's understanding and interpretation. These include a “long_name” for unique descriptions, “units” conforming to unit conventions, and a “missing_value” to represent no data. A “standard_name” attribute, following the Climate Forecast (CF) standard, is assigned when applicable. The final section in ARM NetCDF files consists of global attributes containing information related to the platform's location, time interval, calibration procedures (if available), and contact information for instrument mentors or principal investigators.

A final merged product (aafmerged.c1) was created to provide users with all G-1 airborne measurements in a single file. This merged data product is produced using the ARM Data Integrator (ADI, https://doi.org/10.5439/1999133, Mei and Gaustad, 2024), a framework designed to automate data retrieval, integration, and the creation of time-series NetCDF data products. The ADI allows users to seamlessly combine data products, extract specific variables, and transform them into user-defined coordinate systems. The time dimension of the merged data product aligns with the input aafnaviwg.c1 data stream. All other instrument data were mapped onto this sampling period using ADI's nearest-neighbor transformation method.

All data products produced by the ARM data system (Prakash et al., 2016) adhere to ARM data standards and are made available to the user community via the ARM Data Center in files using the following naming convention (as detailed in Table 4): (sss)(inst)(qualifier)(temporal)(Fn).(dl).(yyyymmdd).
(hhmmss).nc.

For example, a NetCDF file produced for the G-1 airborne deployment at the ACE-ENA field campaign that includes quality-controlled data (in geophysical units) collected starting at 08:31:45 UTC on 18 July 2017, is named enaaafF1.b1.20170718.083145.nc.

All AAF data collected and ingested after 2016 align with the ARM standard data process and format described above. Due to the complex nature of various airborne measurements, some G-1 instrument data have been edited by the mentor (such as the cloud probe) to add additional value. These mentor-edited data are directly ingested from raw data into c1-level data products. For the historical campaign data before 2016, we reprocessed the AAF data following the standard process if the raw data were in a similar format; otherwise, the mentor-edited data were used as the input variables to the ADI process to create the aafmerged.c1 products. Details about individual data points included in the final merged data file for each field campaign are listed in Table S2, as are the primary measurements and their associated standard names used in the recommended aafmerged.c1 product. A “standard_name” is listed in Table S4 for the primary variables, which is consistent with the naming convention based on the CF standard.

ARM includes sufficient metadata in each NetCDF file to facilitate the user's understanding and interpretation. The public data usage rights are as follows: this work is licensed under the Creative Commons Attribution 4.0 International License. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/ (last access: 19 January 2023).

5 Summary

This paper provides an overview of the platform, the aircraft instrumentation, flight tracks, and data collected during the ARM airborne field campaigns and introduces information on data quality control. While numerous studies based on AAF data have both directly and indirectly demonstrated the quality of the datasets, this paper further reinforces this by providing specific examples. It compares in situ measurements with other co-located observations, offering additional evidence to underscore the reliability of the AAF data. A merged dataset containing each flight's meteorological, aerosol, and cloud information was generated for seven AAF field campaigns between 2013 and 2018. The data from 766.4 h of research flights were collected over multiple continents and under various environmental conditions.

Four of the seven field campaigns were based in the USA. One campaign collected data from the wildfires in the US Pacific Northwest and agricultural burns in the lower Mississippi River valley as part of the BBOP in 2013. In 2015, ACAPEX provided data on atmospheric rivers and associated aerosol–cloud interactions that produce heavy precipitation on the US West Coast during the early spring. Research data from ACME V, collected during the summer of 2015, gave scientists insight into trends and variability in trace gases in the atmosphere over the North Slope of Alaska to improve Arctic climate models. In the early summer and autumn of 2016, HI-SCALE provided an extensive dataset geared toward coupled processes that affect the life cycle of shallow clouds through the interaction among aerosol, cloud, the land surface, and ecosystems.

In 2014 (March and October), the airborne sampling moved outside of the USA to Manus in the central Amazon, Brazil, where residential and industrial emissions were extensively characterized by G-1 flights. The GoAmazon2014/15 aircraft campaign data are being integrated with aquatic and terrestrial ecosystem measurements to quantify anthropogenic perturbations to a usually pristine tropical environment. Another international airborne mission was carried out in the eastern North Atlantic region. The ACE-ENA campaign saw the G-1 aircraft fly from Terceira Island in the Azores during the summer of 2017 and the winter of 2018. The campaign studied both seasons to measure key aerosol and cloud processes under various meteorological and cloud conditions with different aerosol sources. The G-1 was then deployed to the Sierras de Córdoba range in central Argentina from October to November 2018 to study orographic convective cloud interactions with their surrounding environment.

The combined observational data from these field campaigns facilitate the study of atmospheric processes, such as boundary layer processes, aerosol–cloud–precipitation interactions, and land–atmosphere–cloud interactions across a wide range of conditions. Although each field campaign faced different challenges and data collection limitations, many previous studies have benefited from the G-1 field campaign data (Gu et al., 2017; Creamean et al., 2018; Fast et al., 2019b; Shrivastava et al., 2019; Berg et al., 2020; Yeom et al., 2021; Zhang et al., 2021; Wang et al., 2023a, b; Zhang et al., 2023), and more manageable data access would further support new users in the research community. By incorporating data from multiple sources, these ARM datasets and open-source tools can provide more accurate and reliable information and assist the model simulation/prediction improvement. Overall, a merged airborne aerosol, cloud, and trace gas dataset covering seven field campaigns is a powerful tool for atmospheric scientists, supporting a more comprehensive understanding of atmosphere processes impacting the climate. We hope our efforts will encourage broader usage of the ARM data and enhance the collaboration between the ARM user facility and the atmospheric science community.