Cloud _ cci AVHRR-PM dataset version 3 : 35 year climatology of global cloud and radiation properties

We present version 3 of the Cloud_cci AVHRR-PM dataset which contains a comprehensive set of cloud and radiative flux properties on a global scale covering the period of 1982 to 2016. The properties were retrieved from Advanced Very High Resolution Radiometer (AVHRR) measurements recorded by the afternoon (post meridiem, PM) satellites of the National Oceanic and Atmospheric Administration (NOAA) Polar Operational Environmental Satellites (POES) missions. The cloud properties in version 3 are of improved quality compared with the precursor dataset version 2, providing better global 5 quality scores for cloud detection, cloud phase and ice water path based on validation results against A-Train sensors. Furthermore, the parameter set was extended by a suite of broadband radiative flux properties. They were calculated by combining the retrieved cloud properties with thermodynamic profiles from reanalysis and surface properties. The flux properties comprise upwelling and downwelling, shortwave and longwave broadband fluxes at the surface (bottom-of-atmosphere BOA) and top-of-atmosphere (TOA). All fluxes were determined at AVHRR pixel level for all-sky and clear-sky conditions, which will 10 particularly facilitate the assessment of the cloud radiative effect at BOA and TOA in future studies. Validation of the BOA downwelling fluxes against the Baseline Surface Radiation Network (BSRN) show a very good agreement. This is supported by comparisons of multi-annual mean maps with NASA’s Clouds and the Earth’s Radiant Energy System (CERES) products for all fluxes at BOA and TOA. The Cloud_cci AVHRR-PM version 3 dataset allows for a large variety of climate applications that build on cloud properties, radiative flux properties and/or the link between them. 15 For the presented Cloud_cci AVHRR-PMv3 dataset a Digital Object Identifier has been issued: https://doi.org/10.5676/DWD/ESA_Cloud_cci/AVHRR-PM/V003 (Stengel et al., 2019). 1 https://doi.org/10.5194/essd-2019-104 O pe n A cc es s Earth System Science Data D icu ssio n s Preprint. Discussion started: 3 July 2019 c © Author(s) 2019. CC BY 4.0 License.


Introduction
Clouds play a critical role for the Earth's weather and climate through their contribution to the Earth's water cycle and their impact on the Earth's energy budget.Clouds impact the energy budget through their interaction with radiation, i.e. clouds usually reflect more solar radiation back to space than the underlying surface and absorb and re-emit infrared (IR) radiation, leading to less IR radiation leaving the system than without clouds.Thus clouds significantly alter important components of the Earth's radiation budget: the shortware and longwave broadband fluxes at the top-of-atmosphere (TOA) and at the surface (bottom-of-atmosphere -BOA hereafter).Analysing cloud coverage and properties, and quantifying the impact they have on the radiation budget is of crucial importance for understanding the Earth's climate and the potential feedback mechanisms in a changing climate.
Since the beginning of the meteorological satellite era at the end of the 1970's attempts have been made to construct global cloud climatologies (e.g.Schiffer and Rossow, 1983) that are of sufficient quality to enable climate studies.Until recently the measurement records of metorological satellite sensors have grown now cover more than 40 years.Even though many difficulties exist when attempting to construct homogeneous and stable climate datasets, those multi-decadal satellite measurements provide the single most important source of measurements with global coverage.Some international efforts exist to regularly improve and extend long-term satellite-based climatologies that contain a comprehensive suite of cloud properties: The Pathfinder Atmospheres -Extended (PATMOS-x, Heidinger et al., 2014), The International Satellite Cloud Climatology Project (ISCCP, Young et al., 2018), The EUMETSAT Climate Monitoring Satellite Application Facility (CM SAF) cloud and radiation data record (CLARA-A2, Karlsson et al., 2016) and the European Space Agency funded Climate Change Initiative ECV Cloud project (Cloud_cci, Stengel et al., 2017).All of these climatologies make use of measurements of the Advanced Very High Resolution Radiometer (AVHRR), which is a passive imaging sensor with 5-6 spectral bands in the visible, nearinfrared and thermal infrared.It is flown on the National Oceanic and Atmospheric Administration (NOAA) Polar Operational Environmental Satellites (POES) and on the EUMETSAT Meteorological operational satellite (Metop) series.There are newer passive sensors in space that also allow for constructing cloud datasets.These are part of research satellite missions by ESA (e.g. the (Advanced) Along-Track Scanning Radiometers on-board the European Remote Sensing Satellite (ERS-1/2) and the Environmental Satellite (Envisat)) and by the National Aeronautics and Space Administration (NASA) (e.g.Moderate Resolution Imaging Spectroradiometer (MODIS) on board the Terra and Aqua satellites).However, mentioned research missions are often characterized by a significantly shorter data record and less spatial coverage due to smaller swath widths.
The MODIS record however has available the combination of cloud properties with high quality TOA radiation measurements made by the Clouds and the Earth's Radiant Energy System (CERES) sensors mounted on board the Terra and Aqua satellites.In addition to the TOA radiation measurements, CERES BOA radiative fluxes are available based on simulations (Kato et al., 2013).Together with available clear-sky fluxes, this set-up provides an excellent basis for analysing the radiative effect of clouds on TOA and BOA energy balances, although the MODIS and CERES records exist only from the year 2000 onwards.Limitations might arise from the coarse spatial resolution of CERES (footprint size of approximately 30 km) and from the fact that the clear-sky fluxes are exclusively based on clear-sky pixels (and interpolation of clear-sky fluxes for gap filling on monthly scales), for which the spatio-temporal sampling and the meteorological conditions are likely to be different.
Consequently, small scale cloud processes and their radiative effects might not be resolved.
The World Climate Research Programme's (WCRP) Global Energy and Water Exchanges (GEWEX) Surface Radiation Budget (SRB) dataset is generated by application of a different approach.Here, retrieved cloud properties are used together with reanalysis information and additional radiative transfer calculations in order to determine all-sky and clear-sky fluxes at the same time for each pixel.The latest release of the GEWEX SRB dataset (v3.0), however, only covers a period until 2007.It makes use of ISCCP DX data, which provides information on a temporal resolution of 3 hours, but includes some deficiencies such as utilizing less spectral information compared to AVHRR-based data, and a relatively coarse spatial resolution.The GEWEX SRB data have been used to revisit the cloud radiative effect on the global scale (e.g.Allan, 2011).
Based on the rationale above it seems logical to construct a record that includes both cloud and radiation properties based on AVHRR, covers a longer time period than alternative records, provides information at finer spatial scales (about 5 km for AVHRR global area coverage -GAC -data) and makes use of all 5 available spectral bands from the visible through the nearinfrared to the thermal infrared.The usefulness of these data is further enhanced by the incorporation of the latest AVHRR intercalibration information and cloud retrieval developments.This paper documents the approaches that have been followed to generate such an AVHRR-based data record with cloud and broadband radiative flux properties, and discusses derived results.The dataset is named Cloud_cci AVHRR-PMv3 (v3 hereafter) and is a successor of AVHRR-PMv2 (v2 hereafter), which contained cloud properties for the period 1982-2014 (see Stengel et al. (2017) for more details), and was already used in numerous studies, e.g. in model evaluation on the global scale (Lauer et al., 2017;Stengel et al., 2018;Eliasson et al., 2018) and on regional scales (Keller et al., 2018;Baró et al., 2018).
Superior to AVHRR-PMv2, AVHRR-PMv3 covers a longer time period , holds cloud properties of improved quality and includes broadband radiative flux properties at TOA and BOA.Appendx A lists additional information about the AVHRR measurement record used.To estimate the radiative fluxes additional radiative transfer calculations were conducted that included additional reanalysis information of tropospheric profiles of temperature and gaseous components as well as surface properties (all interpolated to AVHRR temporal and spatial resolution).This approach is similar to the GEWEX SRB data, thus the retrieved cloud properties are ingested into the reanalysis profiles to represent real clouds with realistic properties at the correct time and place.This is considered a superior approach compared to using reanalysis (thus modelled) clouds directly.All of this information is then input to calculate the broadband fluxes.Although a considerable amount of reanalysis data is still required, this approach provides a means for quantifying the impact of true (retrieved) cloud properties on radiative fluxes at TOA and BOA in a realistic way.This also enables the collection of clear-sky fluxes at the same temporal frequency as all-sky fluxes as opposed to collecting and interpolating the clear-sky fluxes into cloudy areas as is done for the CERES datasets.The set of cloud properties included in v3 is identical to v2 and is outlined in the upper part of Table 1.All data are collected on two processing levels: (a) Level-3U which represents daily composites of non-averaged data collected on a global latitudelongitude grid with 0.05 • resolution and (b) Level-3C which represents monthly averages and monthly histograms on a global latitude-longitude grid with 0.5 • resolution.Input to Level-3U and Level-3C products are pixel-based retrievals using the algorithms described below.Further Level-3U and Level-3C specifications, i.e. the separation of data into liquid/ice sublayers as well as the histograms binning remain identical to v2 (see Tables 4 and 5 of Stengel et al. (2017)).The propagation of derived pixel-level uncertainties into the higher level products Level-3U and Level-3C remains identical to Stengel et al. (2017) as well.

Algorithms
The retrieval system employed for cloud properties is the Community Cloud retrieval for CLimate (CC4CL), which is summarized in Stengel et al. (2017) and described in detail in Sus et al. (2018) and McGarragh et al. (2018).However, further developments have taken place since v2, of which the key elements are listed in the following paragraphs.These improvements are grouped according to the CC4CL subcomponents: cloud masking and cloud phase determination, which now both employ Artificial Neural Network (ANN) schemes and require spectral band adjustments (SBAs), and a component for retrieving the remaining cloud properties using an optimal estimation technique (e.g.Rodgers, 2000).
• Cloud mask: The ANN for cloud detection (ANN mask ) has been retrained using a much larger set of training data, which is composed of collocations between AVHRR measurements and cloud optical depth observed by the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP, Winker et al., 2009).In addition, a different set of channels for daytime conditions, the 3.7 µm channel is now included in the ANN scheme (exception: 1.6 µm is used for NOAA-16 for the period 04/2001 through 04/2003).Table B1 summarizes the ANN mask input data as a function of illumination conditions, while Table B2 reports the empirical thresholds applied posterior to convert the ANN mask output to a binary cloud mask.Downstream, cloud detection is complemented by an additional cirrus test based on 10.8 µm and 12.0 µm IR measurements as defined in Pavolonis et al. (2005).As the cloud detection was developed and fine-tuned for AVHRR on board the NOAA-19 satellites, SBAs are applied for other sensors, which is described in Appendix C. Cloud detection improvements compared to v2 are mainly found for daytime and twilight conditions in general, but in particular also for conditions with snow or ice covered surfaces and in cases of low-level liquid clouds over the sub-tropical and tropical oceans.Validation scores are presented in Section 2.3 reflecting the improvements on the global scale.
• Cloud top phase determination, which in v2 was inferred from the cloud typing procedure of Pavolonis and Heidinger (2004) and Pavolonis et al. (2005), was replaced by a ANN approach for v3 (ANN phase ).The strategy for training the ANN phase was very similar compared to the cloud detection approach, training the ANN phase to emulate CALIOP cloud top phase using AVHRR measurements as primary input data.The exact list of input data for the ANN phase is given in Table B3.Table B4 lists the thresholds applied to convert the ANN phase output to a binary cloud phase.As for cloud Table 1.Cloud_cci AVHRR-PMv3 cloud and radiation properties.ANNmask = artificial neural network for cloud detection, ANNphase = artificial neural network for cloud phase, SV = state vector, PP = postprocessed, PV = Pavolonis algorithm (Pavolonis and Heidinger, 2004;Pavolonis et al., 2005), OE = optimal estimation, BR = BUGSrad (radiative flux algorithm), TOA = top-of-atmosphere, BOA = bottomof-atmosphere (surface), LW = longwave, SW = shortwave.Upper part of the table (cloud properties) has been adopted from Sus et al. (2018).for the cloud phase in v3 compared to v2 when analysing validation results against CALIOP as reported in Section 2.3.
• OE retrieval of cloud properties: The surface reflectance model was revised leading to a corrected handling of the solar zenith angle with most pronounced changes at large angles.Furthermore, bugs were fixed in the code that composes the https://doi.org/10.5194/essd-2019-104 Open Access

Cloud property examples
Figure 1 shows global maps of monthly mean CFC, LCF, COT and CER for June 2014 for v3 Level-3C data -along with the same data from v2.In general, global patterns look very similar with only minor differences between v3 and v2 for CFC and COT.LCF increased (more liquid clouds) from v3 to v2 after a fundamental change of the phase detection approach (see above).CER of v3 is significantly lower than in v2, which is mainly due to fixing a bug in some CC4CL LUTs and introducing alternative single scattering properties, as mentioned in Section 2.1, which only affected retrieved ice cloud properties.

Validation
Cloud_cci AVHRR-PMv3 CMA, CPH and CTH Level-3U products were collocated with equivalent CALIOP products which are assumed to be of superior quality.More specifically, the CAL_LID_L2_05kmCLay-Prov product was downloaded from the ICARE Data and Service Center (http://www.icare.univ-lille1.fr).To investigate the sensitivity of passive imager retrievals to the thinnest cloud layers, the cloud optical depth profiles included in the CALIOP profiles were employed as in Karlsson and Johansson (2013); Stengel et al. (2013); Sus et al. (2018).Following this approach different scenarios for excluding optically thin cloud layers are investigated when discussing validation of CMA, CPH and CTH below.
In addition to the validation against CALIOP, Cloud_cci AVHRR-PMv3 LWP was collocated with AMSR-E observations of LWP (Wentz and Meissner, 2004), and IWP was collocated to DARDAR (raDAR/liDAR Delanoë andHogan, 2008, 2010) observations of IWP.Passive microwave observations of AMSR-E over ocean and active observations of CALIOP and Cloudsat in DARDAR are assumed to provide best reference data for LWP and IWP on global scales.All validation results are accompanied by the equivalent results for v2.
Table 2 reports the validation results for CMA for two scenarios: (1) considering all CALIOP reference pixels as cloudy for which the CALIOP COT is above 0.0 (COT thres =0.0), and (2) considering only those CALIOP reference pixels as cloudy for which the CALIOP COT is above 0.15 (COT thres =0.15).The latter scenario is added to account for the lack of sensitivity of AVHRR measurements to very optically thin clouds.For both scenarios, the scores are generally better for v3 than for v2.Heidke Skill Scores (Heidke, 1926), hitrates and probabilities of detections (PODs) are higher (thus better).The only degradation in the scores is found for the bias, which is slightly more negative in v3 compared to v2.
Table 3 reports the validation results for CPH, also for two scenarios:  Smaller ice particles absorb less radiation coming from below the cloud putting the cloud lower in the atmosphere in the retrieval.In contrast to the bias, standard deviations are reduced for v3 amounting to 2.36 km compared to 2.51 km in v2.
For COT lev =0.15 and COT lev =1.0, very similar findings are made with both of these scenarios showing the reduction in bias and standard deviation with increasing COT lev for ice clouds.This highlights the difficulties in correctly placing (vertically) optically thin clouds and cloud layers when using AVHRR measurements.
Table 5 reports the validation results for LWP.Although the bias for v3 remains small when compared against AMSR-E, it is slightly increased compare to v2 from -1.9 g m -2 to -3.2 g m -2 .Standard deviations are slightly decreased for v3 (26.4 g m -2 ) compared to v2 (27.1 g m -2 ) and the correlation remains unchanged at 0.65.
Table 6 reports the validation results for IWP.The AVHRR-PM IWP generally shows an underestimation of IWP when DARDAR is considered as reference.This underestimation has increased for v3 as the bias has become larger and negative (-307.1 g m -2 for v3 compared to 33.3 g m -2 for v2).However, the standard deviation has decreased significantly from 1299.8 g m -2 to 900.9 g m -2 along with a clear increase in correlation from 0.42 to 0.63.
Despite the assumption that the reference data used is of higher quality than the Cloud_cci data, uncertainties and inaccuracies remain in the reference data as well, which should be kept in mind when interpreting the presented validation scores.
However, summarizing the discussion above, the cloud properties included in AVHRR-PMv3 are considered to be of superior quality than the precursor version.In addition to the cloud properties described in the previous section, radiative broadband flux properties (shortwave and longwave) at TOA and BOA, and for all-sky and clear-sky conditions, were calculated employing the BUGSrad scheme (Stephens et al., 2001, more details below).Furthermore, the photosynthetically active radiation was determined.A full list of radiation properties is given in the bottom part of Table 1.As for the cloud properties, all radiation properties are derived at pixel level, sub sampled to daily, global composites (Level-3U products) and aggregated to monthly Level-3C products.

Algorithm
BUGSrad uses a two-stream approximation along with correlated-k distribution methods for atmospheric radiative transfer (Fu and Liou, 1992).It has been used to investigate aerosol-cloud interactions (Christensen et al., 2017) and to assess the Earth's energy budget using CloudSat observations (Stephens et al., 2012).BUGSrad is applied to a single column, planeparallel atmosphere with ingested cloud properties (i.e.CER, COT, CTP) previously retrieved with CC4CL (see Section 2.1).
BUGSrad uses 18 spectral bands in the electromagnetic spectrum (6 in the shortwave and 12 in the longwave) to compute For longwave radiation, a diurnal cycle correction is applied based on a cosine fit to an observed mean diurnal cycle by applying CC4CL to geostationary Spinning Enhanced Visible and Infrared Imager (SEVIRI).The observed diurnal cycle is converted to a correction factor, which itself is a function of local observation time, to mimic a 24 hour mean.

Radiation property examples
Figure 3 shows examples of Cloud_cci AVHRR-PMv3 Level-3C data of SWF up TOA , LWF up TOA , SWF down BOA , LWF down BOA for all-sky and clear-sky conditions for June 2014.As a general description of these properties, high clearSWF up TOA is found in regions with high surface albedo while high values in SWF up TOA are additional visible in regions with high cloud fraction, and vice versa.ClearSWF up TOA and SWF up TOA depend on incoming solar flux, which, in the month of June, is highest in the tropics and Northern Hemisphere.ClearLWF up TOA is highest in regions with high surface temperatures and low water vapour amounts in the atmospheric column above.Higher water vapour loadings and in particular frequent occurrence of cold clouds significantly reduce the LWF up TOA , for example visible in the tropics and the mid-latitudes.SWF down BOA represents the downwelling solar radiation that is neither reflected nor absorbed by clouds or the atmosphere, thus is, roughly speaking, high where SWF up TOA is low and vice versa.SWF down BOA and clearSWF down BOA strongly depend on illumination conditions.LWF down BOA represents the downwelling radiation emitted by the atmosphere and clouds and is high in regions with high water vapour amounts and further increased when clouds are frequently present.
The product portfolio for radiative fluxes is complemented by SWF down TOA : the incoming solar radiation at the top of the atmosphere and SWF up BOA (LWF up BOA ): the reflected solar (emitted terrestrial) radiation at the Earth's surface (not shown).

BOA radiative fluxes
The Cloud_cci AVHRR-PMv3 BOA radiative fluxes SWF down BOA and LWF down BOA were compared against ground-based reference stations of the World Radiation Monitoring Center (WRMC) Baseline Surface Radiation Network (BSRN, Driemel et al., 2018).For this, monthly mean BSRN SWF down BOA and LWF down BOA values were calculated per station from all available observations and then compared to the nearest neighbouring Cloud_cci grid box. Figure 4 shows scatter plots for all monthly pairs found within the period of 2003 to 2016.The validation scores are reported in  In general, the agreement of the Cloud_cci SWF down BOA , LWF down BOA and LWF up BOA with the BSRN stations is remarkable when considering that only one satellite sensor is used at a time, thus for many locations on Earth only two satellite overpasses (one daytime, one night-time) within 24 hours provide observations.The results are a confirmation that the developed and applied diurnal cycle correction works well, which is more important for the shortwave than for the longwave fluxes.
In contrast, for SWF up BOA more scatter is found in the comparisons to BSRN.Considering SWF up BOA is simply the SWF down BOA multiplied with the surface albedo, and the good validation results for SWF down BOA , this leads to the conclusions that either imperfect surface albedo was used in Cloud_cci, or, more likely, the difference in spatial scales might be the dominating source of the discrepancy found.Fine scale inhomogeneities in surface albedo in the vicinity of the BSRN stations will propagate into the results.
In addition to the BSRN stations, Cloud_cci BOA downwelling and upwelling fluxes were compared to the Clouds and the Earth's Radiant Energy System (CERES) Energy Balanced and Filled (EBAF) surface flux product (Kato et al., 2013), i.e. by means of comparing multi-annual mean maps for the period 2003-2016 (Figures 5 and 6), with corresponding latitude-weighted global mean values given in Table 8.
The Cloud_cci multi-annual mean maps of SWF down BOA for the chosen period agree very well with the CERES products (panels a and b of Figure 5), also for the clear-sky fluxes (panels c and d of Figure 5).This is also supported by global mean values  8 in which Cloud_cci is slightly biased high (+0.9W m -2 for SWF down BOA and +2.2 W m -2 for clearSWF down BOA ).Clear-sky fluxes in both products are mainly characterized by larger incoming solar radiation at the equator, scattering and absorption by atmospheric gases and aerosols, and the surface reflectivity and emissivity.The presence of clouds usually leads to a significant reduction of SWF down BOA locally being a function of optical thickness and cloud fraction over larger domains.The fact that the all-sky fluxes SWF down BOA agree very well with CERES validates the Cloud_cci cloud detection and corresponding cloud property retrievals, which thus can be assumed to be of high quality.
The Cloud_cci multi-annual mean maps of LWF down BOA (Figure 6) also agree well with CERES in terms of global patterns.The absolute values however show systematically higher values for Cloud_cci of about 8 to 9 W m -2 for both all-sky and clear-sky values.In relative terms the systematic differences amount to approximately 2 to 3 %.However, these differences lie within the expected range of the CERES accuracy (Rutan et al., 2015).
The Cloud_cci multi-annual mean maps of SWF up BOA exhibit larger systematic deviations (not shown) than for SWF down BOA .The larger standard deviations retrieved form the solar reflected radiation is primarily related to variances in surface albedo and cloud cover which tend to have significant annual cycles.Global mean values reported in Table 9 give negative biases of -2.7 and -4.6 W m -2 for Cloud_cci which in relative terms correspond to negative deviations of more than 10 %.It remains uncertain which of the two products are more realistic as no real ground truth is available for SWF up BOA that represents spatial scales of satellite pixels (several kilometres).Repeating the validation of SWF up BOA against BSRN but using CERES gives comparable, large deviations (not shown) as found for Cloud_cci (see above).This is in agreement with findings of (Kratz et al., 2010) who reported systematic deviations between CERES and surface observations of SWF up BOA depending on time of day, meteorological condition and location.
Cloud_cci multi-annual maps of LWF up BOA are again closer to CERES (not shown).Global mean values (Table 9) deviate by approximately 2 W m -2 only with larger values for Cloud_cci.In relative terms the differences are about 0.5 %.The Cloud_cci TOA radiative fluxes SWF up TOA and LWF up TOA were compared against the CERES EBAF TOA Edition-4.0data (Loeb et al., 2018).As for the BOA fluxes the comparison includes multi-annual mean maps for the period 2003-2016.Figure 7 shows the maps for SWF for all-sky and clear-sky (clearSWF up TOA ) conditions.Cloud_cci global patterns are very similar to those of the CERES products.High SWF up TOA values are found in regions with high surface albedo, e.g.deserts and polar regions, or with high cloud frequency, e.g. in mid-latitude storm track regions in both hemispheres, in the inner-tropical convergence zone and in regions with persistent marine stratocumulus clouds.Most prominent regions with low SWF up TOA values are the subtropical subsidence regions (low cloud frequency) over the ocean (low surface albedo).It can also be seen that Cloud_cci provides slightly higher values in regions with high SWF (mainly land) and slightly lower values in regions with low SWF (mainly ocean), compared to CERES.The comparisons of the clear-sky fluxes give very similar results.The global mean values given in Table 10 reveal differences of 2.9 and -3.3 W m -2 for all-sky and clear-sky fluxes, respectively.The smaller values in Cloud_cci clear-sky is partly explained by the differences already found for SWF up BOA (see previous section).Figure 8 shows the results of an equivalent analysis for upwelling LWF at TOA.The global Cloud_cci patterns are again in very good agreement to CERES.High LWF up TOA are mainly found in tropical and subtropical regions (high surface temperature) with low cloud frequency or in regions with mainly low level clouds (marine stratocumulus or trade cumulus regions), where the cloud top temperatures are relatively warm.On the contrary, LWF up TOA is low in regions with cold surfaces (e.g.polar regions) and regions with a high frequency of cold clouds.Cloud_cci clearLWF up TOA are dominated by surface temperatures, thus decreasing towards higher latitudes; generally showing a very good agreement to CERES.The values given in (Table 10) reveal slightly smaller global mean values for Cloud_cci compared to CERES.This difference is almost doubled when considering clear-sky fluxes, which is likely due to different sampling approaches.While for Cloud_cci all conditions are included (but removing the clouds when existent), CERES clear-sky TOA fluxes are determined by including clear-sky conditions only.This has the potential to bias TOA longwave fluxes high as clear-sky conditions have less water vapour (Sohn et al., 2010).show a slight reduction in standard deviation for v3 from 27.1 g m -2 to 26.4 g m -2 , accompanied by a slight increase in bias from -1.9 g m -2 to -3.2 g m -2 .Correlations remain unchanged at 0.65.Ice water path validation shows reductions of standard deviations for v3 from 1299.8 g m -2 to 900.9g m -2 compared to v2 (reduction by 30 %).While the clearly increased correlation coefficient emphasises the improvement in v3 as well, the biases are somewhat larger in v3 compared to v2.
A new contribution to version 3 was the addition of top and bottom-of-atmosphere broadband radiative fluxes.Validation of v3 monthly mean downwelling radiative fluxes at BOA against BSRN stations reveals a very good agreement with low standard deviations of 13.8 W m -2 for shortwave and 11.5 W m -2 for longwave fluxes and correlation coefficients above 0.98 for both.
While the bias for shortwave is small (1.9 W m -2 ), a somewhat larger positive bias is found for longwave (7.6 W m -2 ) which is mainly driven by moderate overestimations of larger flux values, but can potentially also partly be due to overestimations in the reference (BSRN).
Cloud_cci AVHRR-PMv3 dataset is summarized.The following Section 2 reports recent cloud retrieval developments and updates, shows product examples and presents validation results all incorporating equivalent results from the precursor dataset version (v2).Section 3 introduces the radiative flux properties and the algorithms they are based on, and, as for cloud properties, presents product examples and evaluation results.Section 4 gives a summary.https://doi.org/10.5194/essd-2019started: 3 July 2019 c Author(s) 2019.CC BY 4.0 License.
Discussion started: 3 July 2019 c Author(s) 2019.CC BY 4.0 License.look-up tables (LUTs) based on pre-calculated radiative transfer simulations.In particular the LUTs for channels with solar reflectance contribution changed considerably.This led to smaller CER retrievals for 3.7µm measurements, in particular for CER ice .Introducing the utilization of the ice cloud single-scattering properties of Baum et al. (2014) (Baran et al. (2005) used before) further reduced the CER ice .For AVHRR-PMv3 cloud optical properties are also retrieved during night-time, facilitated by a differential sensitivity of the radiation in the spectral bands 3.7µm and 10.8µm/12.0µm to COT and CER.Night-time COT and CER retrievals are considered as experimental products and only included in Level-3U products.All retrieved cloud properties are input to the calculation of the radiative fluxes as described in Section 3. As for v2, retrievals of COT and CER are used in v3 to determine LWP and IWP following Stephens (1978).

Figure 2
Figure2presents the same comparison for CTP, LWP, IWP and CLA 0.6 .Global patterns remain very similar again.Mean CTP is lower in v3 than in v2 in the Tropics, which is predominantly due to detecting more very-low level clouds above tropical oceans.While LWP remains similar in v3 compared to v2, IWP is significantly lower in v3 due to lower CER ice (input to the IWP calculation).Unrealistically high LWP and IWP values in polar regions are reduced in v3 due to reduced CER.CLA 0.6 is slightly higher in v3 compared to v2 although the changes are relatively small.Detailed validation was carried out for all cloud properties for which accurate reference data exist.The results of those efforts are presented in the next section, highlighting the quality of the v3 data.

Figure 1 .
Figure 1.Examples of Level-3C (monthly means) Cloud_cci AVHRR-PMv3 data for cloud fraction CFC (a), liquid cloud fraction LCF (b), cloud optical thickness COT (c) and cloud effective radius CER (d).Same data is shown for v2 (e-h).All data is from June 2014.

Figure 2 .
Figure 2. As Figure 1 but for CTP, LWP, IWP and CLA (1) using the cloud phase at the top of the uppermost cloud layer detected by CALIOP as reference (COT lev =0.0), and (2) using the cloud phase at an optical depth of 0.15 into the cloud (top-down) as reference (COT lev =0.15).Comparing the HSS score as overall measure for the correct cloud phase detection, v3 performs better than v2.This is mainly caused by an improved identifications of liquid clouds which comes at the cost of a slightly reduced POD for ice clouds.The liquid bias has increased for v3.Removing the thinnest cloud layers, https://doi.org/10.5194/essd-2019-104started: 3 July 2019 c Author(s) 2019.CC BY 4.0 License.thus accounting for the AVHRR sensor limitation, the improvement of v3 over v2 becomes even clearer.In this scenario, the cloud phase of 84.7 % of all clouds is correctly identified in v3 (according to hitrate scores).It is important to note that the CALIOP data used for validation of cloud detection and cloud phase determination excluded the data that was used for training the ANNs.

Table 4 .
Cloud-top height (CTH) validation results for Cloud_cci AVHRR-PMv3 when compared against CALIOP.Validation results for AVHRR-PMv2 are also reported.Validation measures are standard deviation (Std) and bias.In addition the number of collocated pixels is given.All scores are separated into liquid and ice clouds (both Cloud_cci dataset and CALIOP had to agree on phase) and into three cloud optical depth levels (COTlev) representing at which top-down COT into the cloud the reference CALIOP CTH was taken.
the broadband fluxes.Atmospheric profiles for temperature and water vapour are taken from ERA-Interim.Visible and nearinfrared surface albedo are based on spatiotemporally resolved MODIS climatologies -with all data being identical to the usage in CC4CL.Total solar irradiance is based on SOHO (Solar and Heliospheric Observatory) and SORCE (SOlar Radiation and Climate Experiment) measurements acquired from http://disc.sci.gsfc.nasa.gov/SORCE/data-holdingsusingSOR3TSID_v017and further processed by applying a bi-linear interpolation followed by a bias correction to SOHO measurements to match SORCE.For well-mixed radiatively important trace gases constant values are used (CH 4 = 1.8 ppm, N 2 O = 0.26 ppm).For CO 2 a linearly time dependent concentration is used anchored at 380 ppm for the year 2006.To account for the effect of aerosols on the radiation, an aerosol optical depth of 0.05 was added to the extinction throughout the atmosphere.It is acknowledged that this value is under-representing heavy aerosol loadings which motivates the utilization of spatiotemporally resolved aerosol information for future dataset versions.Due to the angular dependence of the solar illumination together with the low sampling frequency of a single polar-orbiting AVHRR sensor, an angular dependence correction is applied to the shortwave radiation properties to make the data represent https://doi.org/10.5194/essd-2019diurnalcycle of SZA is then used to rescale the incoming and reflected solar radiation and the atmospheric path length for a given set of time stamps throughout the local day.Averaging these samples gives a suitable approximation for a true 24 hour mean, which is needed to determine true climatological means.This procedure is however only applied for Level-3C products, while Level-3U products hold the instantaneous, uncorrected fluxes representing the solar illumination at the pixel location and at the time of observation.

Figure 3 .
Figure 3. Examples of Level-3C (monthly means) Cloud_cci AVHRR-PMv3 data of SWF up TOA , LWF up TOA ,SWF down BOA , LWF down BOA (a-d).clearSWF up TOA , clearLWF up TOA ,clearSWF down BOA , clearLWF down BOA are shown in (e-h) representing the same fluxes as in (a-d) but for clear-sky conditions.All data is from June 2014.

Figure 4 .
Figure 4. Comparison of Cloud_cci bottom-of-atmosphere (BOA) shortwave (SW, panel a) and longwave (LW, panel b) downwelling fluxes with ground-based reference measurements taken at globally distributed Baseline Surface Radiation Network (BSRN) sites for which equivalent reference data was available.Panels (c) and (d) are as (a) and (b) but for upwelling fluxes.Shown are all monthly data pairs within the period 2003-2016.
Comparisons of v3 multi-annual mean values of upwelling and downwelling fluxes at BOA and TOA with CERES additionally emphasises the good quality of the Cloud_cci radiative fluxes in terms of relative spatial pattern and absolute values.Concerning the latter, global mean values of Cloud_cci agree with CERES within 3.3 % for downwelling fluxes at BOA with larger deviations found for longwave fluxes.In contrast, Cloud_cci upwelling longwave fluxes at BOA agree very well with CERES (below 0.5 %), upwelling shortwave fluxes at BOA show deviations up to about 15 %, although the absolute differences are only 4.6 W m -2 at maximum.It however remains uncertain to which extent uncertainties in CERES products contribute to these deviations.In contrast to the BOA, CERES products for TOA fluxes are mainly based on observational information, thus provide an excellent reference for validation.For all-sky fluxes Cloud_cci agrees to CERES within 3% for global mean values.The differences are increased when considering clear-sky fluxes.It is likely that the different approaches to estimate the mean clearsky fluxes in Cloud_cci (including all conditions, but removing the clouds) and CERES (including only cloud-free conditions) contribute considerably to these differences.In summary, Cloud_cci AVHRR-PMv3 represents a dataset of consistent cloud properties and radiative fluxes, which in many aspects is superior to the precursor version v2 as data quality was improved, the product portfolio extended and the covered time period prolonged.Cloud_cci AVHRR-PMv3 offers a large variety of applications including climatological analyses of cloud properties and radiative fluxes as well as their dependency to each other at time scales of several decades.5 Data availabilityFor the presented dataset (Cloud_cci AVHRR-PMv3) a DOI has been issued: https://doi.org/10.5676/DWD/ESA_Cloud_cci/AVHRR-PM/V003(Stengel et al., 2019), of which the landing page points to additional documentation and data download sites.A parallel dataset based on AVHRR on board the NOAA and EUMETSAT morning satellites exists (AVHRR-AMv3), for which a DOI has been issued as well: https://doi.org/10.5676/DWD/ESA_Cloud_cci/AVHRR-AM/V003.The CC4CL retrieval system used to produce the data is version controlled and accessible at github: https://github.com/ORAC-CC/orac/wiki.The LUT response functions (SRF) of AVHRR channels 0.6 µm, 0.8 µm, 10.8 µm and 12.0 µm the SCIAMACHY and IASI measurements were convolved to mimic synthetic AVHRR measurements in each footprint of the considered SCIAMACHY and IASI orbits.Using this procedure for all AVHRR sensors (the AVHRR SRFs differ among the individual satellites) and collecting the synthetic AVHRR measurements in all considered footprints of SCIAMACHY and IASI, a database was composed allowing for linearly fitting all AVHRR sensors to AVHRR onboard NOAA-19.This SBA is applied prior to the application of the cloud https://doi.org/10.5194/essd-2019s)2019.CC BY 4.0 License.

Table 4
reports the validation results for CTH.The validation is stratified by the phase of the cloud and by optical depth into the cloud (top-down) at which the reference CTH is taken from the CALIOP profile.In addition to COT lev of 0.0 and 0.15, COT lev of 1.0 is also included.Generally only few changes in validations scores are found between v3 and v2.While for liquid clouds the scores remain nearly the same, a small degradation in the CTH bias for ice clouds is found.The underestimation of CTH is stronger in v3 compared to v2.For example for the geometrical CTH from CALIOP (COT lev =0.0) the bias degrades from -2.59 km to -3.54 km.One reason for this can be that the LUT-related bug fixes (see Section 2.1) led to smaller CER ice .

Table 2 .
Cloud mask (CMA) validation results for Cloud_cci AVHRR-PMv3 when compared against CALIOP.Validation results for AVHRR-PMv2 are also reported.Validation measures are the probabilities of detecting cloudy and clear scenes (Hitrate, PODcloudy, PODclear) and bias.In addition, the number of collocated pixels is given.The scores are separated into two cloud optical thickness thresholds (COTthres) reflecting above which CALIOP COT the CALIOP pixel was classified cloudy.

Table 3 .
Cloud phase (CPH) validation results for Cloud_cci AVHRR-PMv3 when compared against CALIOP.Validation results for AVHRR-PMv2 are also reported.Validation measures are the probabilities of detecting liquid and ice phase (Hitrate, PODliq, PODice and bias of liquid cloud occurrence).In addition the number of collocated pixels is given.The scores are separated into two cloud optical depth levels (COTlev) representing at which top-down COT into the cloud the reference CALIOP CPH was taken.

Table 5 .
Liquid water path (LWP) validation results for Cloud_cci AVHRR-PMv3 when compared against AMSR-E for the year 2008.Validation results for AVHRR-PMv2 for the same time period are also reported.Validation measures are standard deviation (Std), bias and correlation.In addition the number of collocated pixels is given.

Table 6 .
Ice water path (IWP) validation results for Cloud_cci AVHRR-PMv3 when compared against DARDAR for January to July 2008.
Validation results for AVHRR-PMv2 for the same time period are also reported.Validation measures are standard deviation (Std), bias and correlation.In addition the number of collocated pixels is given.

Table 7 .
An excellent agreement of the Cloud_cci with the reference BSRN measurements is found for both SWF down BOA and LWF down BOA products with correlations above 0.98.Standard deviations are 13.8 W m -2 for SWF down BOA and 11.5 W m -2 for LWF down BOA .The comparisons further reveal positive biases https://doi.org/10.5194/essd-2019-104

Table 7 .
Validation results for monthly Cloud_cci AVHRR-PMv3 shortwave and longwave, downwelling and upwelling radiative fluxes at bottom-of-atmosphere (BOA) when compared against Baseline Surface Radiation Network (BSRN) sites within the period 2003-2016.Validation measures are standard deviation (Std), bias and correlation.In addition the number of data pairs is given.

Table 9 .
As Table8but for BOA upwelling broadband fluxes.