Marine air temperature measurements on-board ships during the daytime are known to be affected by the heating of the ship or platform by the sun. This results in a positive bias during daylight and early night-time hours. The bias varies with sunlight strength or cloudiness (and thus also latitude), relative wind speed, and the size and material of the ship. This solar-heating bias affects both the wet-bulb and dry-bulb temperature measurements, but, as noted by Kent and Taylor (1996), the ships do not act as a source of humidity or change the humidity content of the air. As a result, biases in the specific humidity and dew point temperature due to the solar-heating errors will be negligible. However, care needs to be taken with relative humidity because estimates of the saturation vapour pressure from the uncorrected dry-bulb air temperature will be too high, leading to an underestimate in relative humidity. Ideally, relative humidity should be estimated using the corrected dry-bulb temperature to calculate the saturation vapour pressure and uncorrected wet- and dry-bulb temperature or dew point temperature to calculate the vapour pressure. Previously, efforts have been made to bias-adjust the air temperature observations for solar heating by modelling the extra heating over the superstructure of the ship, taking account of the relative wind speed, cloudiness, time of day, time of year, and latitude (Kent et al., 1993; Berry et al., 2004; Berry and Kent, 2011). These adjustments are complex, and so we have decided not to attempt to implement them for our first version of a marine humidity product given the wide variety of other issues we have accounted for. We have, however, produced daytime, night-time, and combined products to investigate differences that may be caused by the solar-heating bias. Later versions of HadISDH.marine that apply bias corrections for solar heating may reduce the number of daytime data removed.

Humidity measurements can be made in a variety of ways. Instruments can be housed in a screen with ventilation slats, with or without additional artificial aspiration, or handheld in a sling or whirling psychrometer. There is information on instrument ventilation provided up to 2014. Approximately 30 % of ship observations have information in 1973, peaking at ∼75 % by the mid-1990s, as summarized in Fig. 2. Initially, slings were more common for the hygrometer and thermometer, but by 1982 a screen was more common. There is a tendency for the screened instruments, in the absence of artificial aspiration, to give a wet-bulb reading that is higher relative to the slings or whirling instruments where airflow is ensured by the whirling motion. Bias adjustments have been applied to unaspirated humidity observations by Berry and Kent (2011), building on previous bias adjustments of Josey et al. (1999) and Kent et al. (1993). They have also estimated the uncertainty in the bias adjustments. We implement a modified version of their method of bias adjustment for the unaspirated observation types (Sect. 3.3.1) and uncertainty estimation. Uncertainties from instrument bias adjustments will have some spatial and temporal correlation structure as the ships move around (Kennedy et al., 2011a).

Over time there has been a general trend for ship heights to increase. Kent et al. (2007, 2013) quantified the increase from an average of ∼16 m in 1973 to ∼24 m by the end of 2006. Instrument height information is available for some ships between the period of 1973 and 2014, providing heights for the barometer (HOB), thermometer (HOT), anemometer (HOA), and visual-observing platform (HOP). Figure 3 shows the availability of height information and the mean and standard deviation of heights per year in each category for the ship observations selected here. Similar to the ventilation metadata, height information availability is low in 1973, peaking mid-1990s to 2000 and then declining slightly. Prior to 1994 only the platform height was available from WMO Publication 47. This was replaced in 1994 by the barometer height and augmented with the thermometer and visual-observing heights from 2002 onwards (Kent et al., 2007). Anemometer heights have been available from WMO 47 since 1970. All four types of heights increase over time. We conclude that the mean height based on HOP, HOB, and HOT increases from 17 m in 1973 to 23 m by 2014, which differs slightly to that in Kent et al. (2007). If uncorrected, this likely leads to a small artificial decreasing trend in air temperature and specific humidity as, in general, these variables decrease with height away from the surface. The effect on relative humidity is less clear and depends on the relative effects on air temperature and specific humidity.

Prior studies (e.g. Berry and Kent, 2011; Berry 2009; Josey et al., 1999; Rayner et al., 2003; Kent et al., 2013) have applied height adjustments to the air temperature, specific humidity, and wind speed measurements to adjust the measurements to a common reference height and minimize the impact of the changing observing heights on the climate record. These have been based on boundary layer theory and the bulk formulae using the parameterizations of Smith (1980, 1988). In the absence of high-frequency observations of meteorological parameters for each observation location, allowing direct estimation of the surface fluxes, parameterizations have to be made, and an iterative approach is necessary to estimate a height adjustment (Sect. 3.3.2). We have followed these previous approaches and estimated height adjustments for all observations and variables of interest. Where observing heights are unavailable, we have made new estimates (Sect. 3.3.2). We have also provided an estimate of uncertainty on these height adjustments, which are larger where we have also estimated the height of the observation. The uncertainties from height adjustments will have some spatial and temporal correlation structure.

Recording and reporting formats and practices have changed many times over the 20th century, affecting the climate record. Some formats required the wet-bulb temperature to be reported, others the dew point temperature, and some allowed either or both (https://www.wmo.int/pages/prog/amp/mmop/documents/publications-history/history/SHIP.html, last access: June 2019). Some earlier formats restricted space to reporting temperature to whole numbers only, and this practice has continued, with some ships continuing to report the dew point (or wet-bulb) temperature and sometimes even the dry bulb temperature to whole numbers. A practice of truncation of the dew point depression has been noted for the pre-1982 data (http://icoads.noaa.gov/corrections.html, last access: June 2019), which would result in spuriously high humidity (both in relative and actual terms). It is clear from the ICOADS3.0.0 and 3.0.1 data that there has been a practice of reporting values to whole numbers rather than decimal places, both for air temperature and dew point temperature. Rounding dew point temperature and air temperature could result in a ±0.5 ∘C error individually or a just less than ±1 ∘C error in dew point depression for a worst-case-scenario combination.

Whole-number reporting is an issue throughout the record for both variables; a breakdown of air and dew point temperature by decimal place over time is shown in Fig. S1 in the Supplement. Air temperature also shows a disproportionate frequency of half degrees (.5s). The percentage of whole numbers (.0s) declines over time, dramatically in the mid- to late 1990s for air temperature and from 2008 for both air and dew point temperature. This decline in the 1990s and in part also the general decline appear to be linked to an increase in numbers of moored buoys (see Fig. 5); a similar analysis without the moored buoys (not shown) shows greater consistency over time. The dew point temperature has two distinct peaks in whole-number frequency in the 1970s and mid-1990s to early 2010s. The latter peak is more pronounced when moored buoys are not included. The early peak is somewhat consistent with the restriction in transmission space prior to January 1982. This was previously thought to have been a possible cause of higher relative humidity over the period 1973–1981 compared to the rest of the record in the HadCRUH marine relative-humidity product (Willett et al., 2008). The pre-1982 moist bias was also apparent in the global marine relative-humidity product of Dai (2006), which like HadCRUH used dew point temperatures. The NOCSv2.0 product preferentially utilizes the wet-bulb temperatures from ICOADS, which are not affected by whole-number reporting to the same extent.

Rounding of temperature alone should not affect the mean dew point temperature, specific humidity, or vapour pressure. However, as with the solar bias issue, it is sensitive to the point at which the reported dew point temperature was derived from the measured wet-bulb temperature or relative humidity. Most likely, this would be done prior to any rounding or truncating for reporting, but during later conversion of various sources into digital archives or corrections the dew point temperature may have been reconstructed (https://icoads.noaa.gov/e-doc/other/dupelim_1980, last access: June 2019). The effect of rounding on a monthly mean grid box average should be small as these errors are random and should reduce with averaging. However, there is a risk of removing very high humidity observations when a rounded dew point temperature then exceeds a non-rounded air temperature. Such values are removed by our supersaturation check (Sect. 3.2). We do not feel able to correct for this issue but instead include an uncertainty estimate for it. Overly frequent whole numbers are identified both during quality control track analysis and deck analysis. This is discussed in more detail in Sect. 3.4. Clearly, there are various issues that can arise linked to the precision of measured and reported data in addition to conversion between different units (e.g. Fahrenheit, Celsius, and kelvin; Fig. S1) and between different variables.

All observations are subject to some level of measurement error, and, outside of precision laboratory experiments, the errors can be significant. The BIPM (International Bureau of Weights and Measures – Bureau International des Poids et Mesures) Guide to the Expression of Uncertainty in Measurement (BIPM, 2008) describes two categories of measurement uncertainty evaluation. A Type A evaluation estimates the uncertainty from repeated observations. A Type B evaluation of the uncertainty is based on prior knowledge of the instrument and observing conditions. Within this study we use a Type B evaluation, adjusting for systematic errors and inhomogeneities due to inadequate ventilation and changing observing heights (screen and height adjustments) and estimate the residual uncertainty. For the random components, we make the conservative assumption that all measurements were taken using a psychrometer (wet-bulb and dry-bulb thermometers), which allows us to follow the HadISDH.land methodology of Willett et al. (2013, 2014) as described in Sect. 3.4. An assessment of the frequency of hygrometer types (TOHs) within our selected ICOADS3.0.0 and 3.0.1 data shows this to be a fair assumption as the vast majority of ships (where metadata is available: ∼30 %, increasing to ∼70 % from 1973 to 1995 then decreasing to 60 % by 2014) are listed as being from a psychrometer (Fig. 4). Electric sensors are becoming more common and made up ∼30 % of observations by 2014 (the end of the metadata information). There are no instrument type metadata for ocean platforms or moored buoys. As it is likely that most buoy observations are made using RH sensors, we plan to develop an RH-sensor-specific measurement uncertainty in future versions.

There are other issues specific to humidity measurements that may be further sources of error. Hygrometers that require a wetted wick (i.e. psychrometers) and thus a source of water are vulnerable to the wick drying out or contamination, especially by salt in the marine environment. The wick drying results in erroneous relative-humidity readings of 100 %rh, where the wet bulb essentially behaves identically to the dry-bulb thermometer. There can also be issues when the air temperature is close to freezing, depending on whether the wet bulb has become an ice bulb or not and whether wet-bulb or ice-bulb calculations are used in any conversions. Humidity observing in low temperatures can be generally problematic. For radiosondes, there has previously been a practice of recording a set low value when the humidity observation falls below a certain value (Wade, 1994; Elliott et al., 1998). It is debateable how likely such low humidity values are over oceans, and this practice has not been documented for ship observations. However, the set-value issue is something to look out for. Wet bulb thermometers (and other instruments) can experience some hysteresis at high humidity, where it takes some time to return to a lower reading. The wet bulb also requires adequate ventilation, which has been discussed above.

These can be accounted for to a large extent through quality control, but some error will inevitably remain. We can increase our confidence in the data by comparison with other available products and general expectation from theory.

We screen all ICOADS data to sub-select only those observations passing the following criteria.

There must be a non-missing T and Td value.

We have not used any of the preset flags from ICOADS processing to ensure methodological independence of HadISDH and a process that allows for exploration and analysis of different methodological choices. The quality control processing employed here largely follows the methodology for HadSST4 (Kennedy et al., 2019), with some changes to the climatology check and buddy check thresholds to increase regional sensitivity and additional humidity-specific checks. A flag for whole-number prevalence has also been added, but this is used for uncertainty estimation and not to remove an observation. All observations have their nearest 1∘×1∘ pentad mean climatology (source depends on iteration – Sect. 3.5) subtracted to create a climate anomaly.

Each observation is passed through a suite of quality control tests, which are summarized in Table 1 along with whether the quality control tests are used to remove or just to flag the observations and the stage of processing at which they are applied. The climatology check differs from the static HadSST3 threshold of climatology for air temperatures of ±8 ∘C. We have allowed for a variable threshold depending on the nearest 1∘×1∘ pentad climatology standard deviation σ. This is set at 5.5σ. It accounts for the lower variability in the tropics and greater variability in the mid-latitudes. We have set minimum and maximum σ values of 1 and 4 ∘C, respectively, resulting in a minimum range of ±5.5 ∘C and a maximum range of ±22 ∘C. Several thresholds were tested, with the selected threshold balancing avoiding acute cut-offs in the data distribution while still removing obviously bad data (Figs. S3 to S6). Given that outliers are assessed by comparing a point observation with a 1∘×1∘ pentad mean, the thresholds have to be relatively large.

The buddy check compares each observation's climate anomaly with the average of the climate anomalies of its nearest neighbours in space and time, expanding the search area in space and time as necessary until at least one neighbour observation is found. The permitted difference is set by the climatological standard deviation of the candidate 1∘×1∘ pentad grid box multiplied by an amount dependent on the number of neighbours present. There are five levels of searches.

±1∘ latitude and longitude and ±2 pentads: the climatological standard deviation is multiplied by 5.5, 5.0, 4.5, and 4.0 for 1–5, 6–15, 16–100, and >100 neighbouring observations, respectively.

Figure 6 shows the final number of observations passing through initial selection and then third-iteration quality control by platform type (PT). The quality control does not significantly affect one platform over another. The performance of these tests is demonstrated for 4 example months in Figs. S3 to S6. These reveal a slight positive bias in the removed air temperature observations and negative bias in removed dew point temperature. Removals in terms of relative humidity and specific humidity similarly tend to have a negative bias. It is clear that the majority of grossly erroneous observations are removed. The change in climatology between iterations of the quality control process (Sect. 3.5) also makes a difference to removals. This is because the observation-driven climatologies do not provide complete spatial coverage and because the ERA-Interim climatologies are cooler and drier than the observations (Sect. 4.1). Removals are dense in the Northern Hemisphere and especially sparse around the tropics. The addition of the buddy check in the third iteration considerably increases the removal rate, noticeably over the Southern Hemisphere and the tropics.

The quality-control flagging rate for the third iteration reduces over time from ∼25 % to ∼18 %, as shown in Fig. S7. This is driven by the buddy check and track check. Proportionally more observations are flagged during the daytime than night-time, but the inter-annual behaviour is very similar. The daytime increase is driven by the larger number of air temperature buddy and climatology check failures. This could be due to the issue of solar heating of the ship structure during the daytime. The main source of test fails by a large margin is the buddy check, followed by the climatology check and track check. There does not appear to be a strong difference in the distribution of removals from each test between the 1973–1981 and 1982–1990 periods that might explain the pre-1982 moist bias (Fig. S8, Sect. 4.2). There is an increase in removals from repeated saturation and supersaturation events over time, particularly the late 2000s. This may be related to the decrease in psychrometer deployment over time and increase in electric and capacitance sensors as shown in Fig. 4. The latter have increased significantly since the mid-2000s.

The whole-number flags show very different behaviour to the other checks and to each other over time in Fig. S7. These depend on the ability to assign each observation to a track or voyage and the frequency of whole-number observations on that voyage; hence, these flags are not a true reflection of the whole-number frequency. Compared to the actual proportion of whole numbers shown in Fig. S1, these tend to exaggerate the annual patterns, but the shape is broadly similar. This method of identifying problematic whole numbers appears to under-sample the true distribution, especially for air temperature pre-1982. An additional deck-based check is applied later for estimating uncertainty from whole numbers (Sect. 3.4).

Note that the NOCSv2.0 dataset, with which we compare our specific-humidity data, includes an outlier check that removes data greater than 4.5 standard deviations from the climatological mean. This test has already been applied within the ICOADS format, and so the NOCSv2.0 excludes any data with ICOADS trimming flags set (Wolter, 1997). We do not use the trimming flags to select data. They also apply a track check based on Kent and Challenor (2006).

Given the issues raised in Sect. 2, it is desirable to attempt to adjust the observations to improve the spatial and temporal homogeneity and accuracy of the data. As discussed in Sect. 2.1, we have not attempted to adjust for solar biases in this first-version product. We have made adjustments for instrument and height biases and estimated uncertainties (summarized in Table 1) in these adjustments.

The availability of machine-readable metadata alongside each observation enables specific adjustment for known biases and inhomogeneities. This differs to the approach for the HadISDH.land dataset, where no substantial digitized metadata currently exist. By necessity, adjustment for biases (inhomogeneities) is done using the Pairwise Homogenization Algorithm (Menne and Williams, 2009). This is a neighbour-comparison-based statistical algorithm to detect change points and resolve the most reasonable adjustments. It is very likely that inhomogeneities that affect the land data such as instrument changes, instrument housing changes, and practice changes also affect the marine data. However, this level of detail is not available in the metadata nor is it straightforward to adjust for even if it were because of the mobile nature of ship data. Although a neighbour-based comparison is possible and useful at the single-observation level (e.g. buddy check), it is not useful in the manner in which it is used for land observations from static weather stations. Arguably, the region-wide biases such as increasing ship heights and ventilation biases are of greater concern for long-term trends than the more ship-specific inhomogeneity owing to instrument or housing changes. We acknowledge that, similar to the land data, there will be inhomogeneity or bias remaining within the HadISDH.marine dataset which we cannot detect or adjust for but argue that we have removed the large errors from the dataset. Future versions will take advantage of greater metadata and statistical tools as they become available.

Three other sources of uncertainty affect the marine humidity data at the observation level. These are measurement uncertainty Um, climatology uncertainty Uc, and whole-number uncertainty Uw. These are all assessed as 1σ standard uncertainties.

We have estimated Um for each observation following the method used for HadISDH.land (Willett et al., 2013, 2014). This assumes that humidity was measured using a psychrometer, which is a reasonable assumption for the marine ship data (Fig. 4). The HadISDH.land measurement uncertainty is based on an estimated standard (1σ) uncertainty in the wet-bulb and dry-bulb instruments of 0.15 and 0.2 ∘C, respectively. As shown in Table S3, the equivalent uncertainty for the other variables depends on the temperature. The uncertainty is applied as a standard uncertainty in RH depending on which bin the air temperature falls in. This is then propagated through the other variables starting with vapour pressure using the equations in Table S1.

Whole numbers of air and/or dew point temperature that have been flagged as such during quality control (Sect. 3.2) or that belong to a source deck or year where whole numbers make up more than 2 times the frequency of other decimal places (Table S4) are given an uncertainty Uw. These decks and years where whole numbers are very common differ for air and/or dew point temperature. Clearly with so many decks affected, the removal of entire decks to remove any whole-number biasing could easily reduce sampling to critically low levels. We cannot distinguish between observations that have been rounded versus those that have been truncated, so we assume that all offending whole numbers have been rounded. This means that the value could be anywhere within ±0.5 ∘C, with a uniform distribution. Hence, where only air or dew point temperature is an offending whole number, the standard 1σ uncertainty expressed in air or dew point temperature (∘C) is 4Uw=0.53. Where both air and dew point temperature are offending whole numbers, the standard 1σ uncertainty expressed in air or dew point temperature (∘C) for dew point depression, relative humidity, and wet-bulb temperature is 5Uw=13. There is uncertainty Uc in the climatological values used to calculate climate anomalies because of missing data over time, uneven and sparse sampling in space, and also the inevitable mismatch between a point observation and a 1∘×1∘ gridded pentad climatology. This uncertainty reduces with the number of observations contributing to the climatology Nobs and with the variability of the region σclim. The climatologies used to create the anomalies have undergone spatial and temporal interpolation to move from 5∘×5∘ gridded monthly climatologies and climatological standard deviations σclim to maximize coverage, and so it is not straightforward to assess the number of observations contributing to each 1∘×1∘ gridded pentad climatology, and the true σclim is likely greater. The minimum number of years required to be present over the 30-year climatology period is 10. Therefore, we assume a worst-case scenario of Nobs=10. Hence, for a standard 1σ uncertainty the following equation applies: 6Uc=σclimNobs.

To create a quasi-global monitoring product, the raw observations need to be gridded. The spatial density is too low for high-resolution grids, and the intended purpose is for this marine product to be blended with the HadISDH.land humidity product, which is on a 5∘×5∘ grid at monthly resolution. Hence, the point hourly observations must be averaged to monthly mean gridded values.

The sparsity of the data means that there is a risk of bias due to poor sampling. A 5∘×5∘ grid box covers an area greater than 500 km2×500km2, which, despite the large correlation decay distances of both temperature and humidity, can include considerable variability. Furthermore, a monthly mean can be made up of a strong diurnal cycle and considerable synoptic variability. This is minimized by the use of climate anomalies, but regardless, care should be taken to ensure sufficient sampling density while maximizing coverage where possible.

Several data-density criteria were trialled to balance spatial coverage and poor representativeness (high variance) of the grid box averages. Climate anomalies are created at the raw observation level by subtracting the nearest 1∘×1∘ pentad climatology (1981–2010), and so we can grid both the actual values and the anomalies. Gridding of the anomalies is safer than gridding actual values in terms of biasing through poor sampling density because the correlation length scales of anomalies are higher than for actual temperatures. Initially, ERA-Interim is used to provide a climatology. This then requires an iterative approach to produce an initial observation-based climatology and improve the climatology through quality control. To reduce biasing further we grid the data in six stages to create an average at each stage. The entire process including quality control, bias adjustment, gridding, and three iterations is shown diagrammatically in Fig. 5 and each gridding stage described below.

Create 1∘×1∘ 3-hourly gridded means of the hourly observations of actuals and anomalies; there must be at least one observation.

The observation uncertainties also need to be gridded and the total observation uncertainty Uo calculated. Ships move around, and so their uncertainties also track around the globe. This means that the uncertainty in any one point or grid box bears some relationship to nearby points or grid boxes over time and space and cannot be treated independently. Correlation needs to be accounted for in both gridding and subsequently creating regional averages from grid boxes to avoid underestimation. The five sources of observation uncertainty are summarized in Table 2. The non-aspirated-instrument-adjustment uncertainty Ui, height adjustment uncertainty Uh, and climatology uncertainty Uc persist over time and space as ships move around. These are accordingly treated as correlating completely within 1 grid box month. The measurement uncertainty Um and whole-number uncertainty Uw are likely to differ from observation to observation, and so they are treated as having no correlation within 1 grid box month. Hence, observation uncertainty sources are first gridded individually, following the first four steps outlined above and taking into account correlation where necessary. For those that do not correlate (Um and Uw), the grid box mean uncertainties Ugb for each source are combined over N points in time and space as follows: 7Ugb=a2+b2…+n2N. For those sources that do correlate (Uc, Ui, and Uh), assuming r=1, the grid box mean uncertainties Ugb for each source are combined over N points in time and space as follows: 8Ugb=a+b…+nN. To create the total observational uncertainty for each grid box, the grid box quantities of the five uncertainty sources can then be combined in quadrature: 9Uo=Uc2+Um2+Uw2+Uh2+Ui2. Given the general sparsity of observations across each grid box month and the uneven distribution of observations across each grid box and over time, there is also a grid box sampling uncertainty component, Us. This is estimated directly at the 5∘×5∘ monthly grid box level and follows the methodology applied for HadISDH.land (Willett et al., 2013, 2014), denoted SE2, which is based on station-based observations from Jones et al. (1997): 10Us=s¯i2r¯1-r¯1+Ns-1r¯, where s¯i2 is the mean variance of individual stations within a grid box, r¯ is the mean inter-site correlation, and Ns is the number of stations contributing to the grid box mean in each month. The mean variance of individual stations within the grid box is estimated as 11s¯i2=S^2NSC1+NSC-1r¯, where S^2 is the variance of the grid box monthly anomalies over the 1982–2010 climatology period, and NSC is the mean number of stations contributing to the grid box over the climatology period. The mean inter-site correlation is estimated by 12r¯=x0X1-exp⁡-x0X, where X is the diagonal distance across the grid box, and x0 is the correlation decay length between grid box means. We calculate x0 as the distance (grid box midpoint to midpoint) at which correlation reduces to 1/e. To account for the fact that marine observations generally move around at each time point, we use the concept of pseudo-stations to modify this methodology. For any one day there could be 25 1∘×1∘ grid boxes, and so we assume that the maximum number of pseudo-stations per grid box is 25, which is broadly consistent with the number of stations per grid box in HadISDH.land. Over a month then, there could be a maximum of 775 1∘×1∘ daily grid boxes contributing to each 5∘×5∘ monthly grid box. Given ubiquitous missing data and sparse sampling, the maximum in practice is closer to 600. Using these values we then scale the actual number of 1∘×1∘ daily grid boxes contributing to each 5∘×5∘ monthly grid box to provide a pseudo-station number between 1 and 25 for each month (Ns) and then the average over the climatology period (NSC).

The grid box Uo and Us uncertainties are then combined in quadrature, assuming no correlation between the two sources. This gives the full grid box uncertainty Uf. Calculation of regional-average uncertainty and spatial coverage uncertainty is covered in Sect. 4.

At the end of each iteration (Fig. 5), observation-based climatology fields are created at both the monthly 5∘×5∘ grid and, by interpolation, pentad 1∘×1∘ grid (Sect. 3.5). These are then used to quality-control and create anomaly values for the next iteration. Hence, the second-iteration quality-controlled data are used to build the final third iteration, and therefore there should be no lasting effect from having used the ERA-Interim fields initially. The quality-controlled, buddy-checked, and bias-adjusted third iteration is used to create the final climatology provided to users.

To compare the use of ERA-Interim versus the observation-based climatology to calculate anomalies and quality-control the data, we show difference maps of the second iteration minus ERA-Interim pentad 1∘×1∘ grid climatologies and climatological standard deviations in Figs. S9 to S14 for a selection of pentads and variables. Note that ERA-Interim fields are for 2 m above the ocean surface, whereas the raw observations range between approximately 10 and 30 m above the surface. In normal conditions we may therefore expect ERA-Interim to provide climatologies that are warmer and moister than the observations. However, overall, ERA-Interim appears drier (both in absolute and relative terms) and cooler than the observation-based climatologies. For humidity this is consistent with the results of Kent et al. (2014). For the majority of grid boxes these differences are within ±2 g kg -1, ±2 %rh, and ±2∘C. However, differences are especially strong around coastlines, with magnitudes exceeding ±10 g kg -1, ±10 %rh, and ±10∘C. This is to be expected given that ERA-Interim coastal grid boxes will include effects from land, especially at the relatively coarse 1∘×1∘ grid resolution. For relative humidity there are more regions where ERA-Interim is more saturated, and there is more seasonality in the differences. Relative humidity is less stable spatially and on synoptic timescales and also more susceptible to biases and errors than specific humidity and air temperature, largely because it is affected by errors in both air temperature and dew point temperature. For temperature, the coastal difference can be positive or negative depending on the season.

The climatological standard deviations are generally lower in the second-iteration observations compared to ERA-Interim. Differences are generally between ±2 g kg -1, %rh, and ∘C, but for relative humidity there are expansive regions in the extratropics to mid-latitudes, especially in the Northern Hemisphere, where climatological standard deviations are up to 5 %rh lower in the observations. The generally lower variability in the observation-based climatology is to be expected given the interpolation from monthly mean resolution and interpolation over neighbouring grid boxes where data coverage is limited. However, much of the tropics, particularly in the Southern Hemisphere, tend to show more variability in the observations. Similarly, many of the peripheral grid boxes (those at the edge of the spatial coverage and therefore more likely to be interpolated from nearby grid boxes rather than based on actual data) show higher variability for specific and relative humidity and lower variability for air temperature. All of these grid boxes are in data-sparse regions, which likely contributes to the higher variability. Ideally, observation-based climatologies would be created directly at the pentad 1∘×1∘ grid, but this severely reduces spatial coverage of the climatology fields and any product based on them. A balance has to be made between coverage and quality.

Annual mean 5∘×5∘ climatologies (no interpolation) from the third-iteration quality-controlled, bias-adjusted ship-only product are shown in Fig. 7 for specific humidity, relative humidity, air temperature, and dew point temperature. These have a minimum data presence threshold of 10 years for each month over the climatology period and at least 9 climatological months present for the annual climatology. Data coverage is virtually non-existent in the Southern Hemisphere below 40∘ S, and Northern Hemisphere coverage diminishes drastically above 60∘ N. These climatologies are as expected for these variables and compare well in terms of broad spatial patterns with ERA-Interim (not shown). There is good spatial consistency considering that no interpolation has been conducted, meaning that any erroneous grid boxes should stand out. We conclude that, as a first-version product, these climatologies look reasonable.

Global-average quantities are key measures of climate change, and so we focus here on the differences arising from the various processing steps of HadISDH.marine along with the NOCSv2.0 specific humidity and ERA-Interim reanalysis products. Global averages have been created by weighting each grid box by the cosine of its latitude at the grid box centre. All time series shown are the renormalized anomalies with a mean of 0 over the 1981–2010 period. Figures 8 to 11 show time series for specific humidity, relative humidity, dew point temperature, and air temperature, respectively. Decadal linear trends (shown) are computed using ordinary least-squares regression with ranges representing the 90th-percentile confidence interval calculated using AR(1) correction (Santer et al., 2008).

For all variables, there are only small differences in the global-average time series between the various processing steps – from the raw data (noQC) to the third-iteration quality-controlled (noBA: no bias adjustment) and then the bias-adjusted data (BA). They are smallest for air temperature and largest for relative humidity, but all steps result in global-average trends that are significant and in the same direction and have similar inter-annual variability. We consider these trends to be significant because the 90th-percentile confidence intervals around the trend are not large enough to bring the direction of the trends into question. The trends in the global average are positive over the 1973–2018 period for specific humidity, dew point temperature, and air temperature and negative for relative humidity. The linear trends for the final HadISDH.marine.1.0.0.2018f version are 0.07±0.02 g kg-1 per decade, -0.09±0.08 %rh per decade, 0.09±0.02 ∘C per decade, and 0.11±0.03 ∘C per decade for specific humidity, relative humidity, dew point temperature, and air temperature, respectively. Hence, we conclude that HadISDH.marine shows moistening and warming since the 1970s globally in actual terms but that the air above the oceans appears to have become less saturated and drier in relative terms. This differs from the theoretical expectation that changes in relative humidity over ocean are strongly energetically constrained to be small, of the order of 1 % K-1 or less, and generally positive (Held and Soden, 2006; Schneider et al., 2010). Model-based expectations also suggest small positive changes (Byrne and O'Gorman, 2013, 2016, 2018). Despite careful quality control and bias-adjustment, the previously noted moist humidity bias pre-1982 is still apparent in the bias-adjusted (BA) data. The linear trend in relative humidity from 1982 to 2018 is -0.03±0.13 %rh per decade and therefore not significantly decreasing, which is more consistent with expectation.

Since there are considerable known issues affecting the marine humidity data and because there are large outliers (Figs. S3 to S6), the effect of quality (noQC compared to noBA) might be expected to be large. Furthermore, approximately 25 %, dropping steadily over time to 18 %, of the initial selection of data has been removed by the quality control (Fig. 5), so there is a considerable difference in the amount of data contributing to the quality-controlled version compared to the raw version. Despite all of this, differences are relatively small. Overall, the quality control makes the positive trends smaller (specific humidity, dew point temperature, and air temperature) and negative trends larger (relative humidity). The effect of quality control, including buddy checking, is largest in the 1970s to early 1980s, when the largest number of data was removed by quality control. This is especially noticeable for relative humidity and dew point temperature, suggesting that the pre-1982 bias, although present to some extent in the raw (noQC) data, could be exacerbated by the quality control. This could be due to erroneous removal of good data, but investigation (Figs. S3 to S8) suggests that much of the data removal was appropriate; many very low relative-humidity values were removed. It could also be an artefact of the reduced number of observations after quality control, reducing the chance of averaging out random error. To explore whether the presence of whole numbers in the record has contributed to the pre-1982 bias, we have processed a bias-adjusted version with all whole-number flagged data (Table 1) removed (BA_no_whole), which is shown against the noQC and BA versions in Fig. 9d. The resulting global-average trend is largest in the BA_no_whole version, even over the 1982–2018 period, and the pre-1982 bias is still clear. We conclude that the pre-1982 moist bias remains apparent in HadISDH.marine and is not yet well understood, and quality control of the pre-1982 data is an area for more research in future versions.

The bias adjustment (BA, BA_HGT, BA_INST) reduces the negative trends in relative humidity compared to the quality-controlled (noBA) data and increases the positive trends in specific humidity and dew point temperature relative to the quality-controlled data. The effect of bias adjustment is negligible for air temperature, which only has adjustment for ship height applied. For the humidity variables the height adjustment has a far larger effect than the non-aspirated-instrument adjustment. The non-aspirated-instrument adjustment makes the positive trends in specific humidity and dew point temperature slightly smaller and the negative trends in relative humidity slightly larger. The height adjustment has the opposite effect. For relative humidity, the bias adjustments appear to have introduced greater intra-decadal-scale variability but retained the inter-annual patterns, again highlighting the sensitivity of relative humidity compared to the other variables. Given that these biases exist we do have to try and mitigate their impact. However, this is a focus area for investigation and improvements in future versions of HadISDH.marine.

The time series that include data from moored buoys compared to those from ships only (“all” versus “ship”) show smaller positive trends for specific humidity and air temperature and larger negative trends for relative humidity. Moored buoys begin to play a role from the late 1980s, increasing in number dramatically to make up over 50 % of the observations by 2015. The “all” time series can be seen to diverge slightly from the “ship” time series in the latter part of the record. Therefore, it is more consistent to produce the final HadISDH.marine version without inclusion of moored-buoy data.

Before quality control there are more daytime ship observations than night-time ship observations in the early record (∼1 000 000 compared to ∼800 000 yr-1), but this evens out by the end of the record to ∼900 000 yr-1. However, the quality control removes more daytime observations than night-time observations, especially in the 1970s and 1980s, such that both contribute ∼700 000 observations per year, dipping in the middle of the record. There has been no bias adjustment for solar heating of ships applied in this version of HadISDH.marine, so the daytime data may contain some artefacts of solar heating. If this is a problem it should affect the air temperature and relative humidity but not the dew point temperature or specific humidity (Sect. 2.1). While the full dataset (both) combines both daytime and night-time data, for various grid boxes and seasons there is only either a daytime or night-time value present. As such, the “both” time series and its linear trend may not be a straightforward average of the “day” and “night” time series and trends. For specific humidity, dew point temperature, and air temperature, the “day” and “night” trend differences are essentially negligible, with linear trends identical or within 0.01 g kg-1 per decade. Even for relative humidity the differences are small. The “day” time series gives the largest negative trend, followed by “both”, which is 0.01 %rh per decade smaller, and then “night”, which is 0.02 %rh per decade smaller again. The negligible differences in air temperature suggest that solar heating is not a significant concern, at least at the global-average scale. Relative humidity is very sensitive to any differences in the data, but even these differences are fairly small and do not change the overall conclusion of decreasing full-period trends and no significant trend over the 1982–2018 period. “Night” trends are often thought to provide a better signal of change because they are generally free from convective and shortwave radiative processes and more a measure of outgoing longwave radiation. The main conclusion here is that trends and variability are very similar in the daytime, night-time, and combined time series, which adds confidence in their representativeness of real-world trends and variability.

In terms of linear trend direction, HadISDH.marine compares well with other monitoring estimates from NOCSv2.0 and ERA-Interim and to other reanalyses and older products (Fig. 1). ERA-Interim in Figs. 8 to 11 is from analysis fields of 2 m air temperature and dew point temperature and has been masked to ocean coverage using a 1∘×1∘ land–sea mask and also to HadISDH.marine coverage for comparison. Note that the ERA-Interim time series shown in Fig. 1 are from background forecast values to avoid biases introduced from ship data and ocean-only points over open sea. Both NOCSv2.0 and HadISDH.marine are estimates of 10 m quantities, and the NOCSv2.0 coverage is similar to that of HadISDH.marine, but it only extends to 2015. NOCSv2.0 shows the largest trends in specific humidity over the 1979–2015 common period, 0.04 g kg-1 per decade greater than HadISDH.marine. The inter-annual patterns are broadly similar but with some differences showing that methodological choices do make a difference given that the underlying observations are from the same source. ERA-Interim shows very weak moistening compared to HadISDH.marine for specific humidity and dew point temperature and slightly weaker warming for air temperature. Over the longer 1979–2018 period, ERA-Interim trends are slightly larger for specific humidity but still weaker than in HadISDH.marine. The decreasing saturation in relative humidity is very strong in ERA-Interim, at more than 2 times the HadISDH.marine trend over the common period. The masking to HadISDH.marine coverage surprisingly makes very little difference in the linear trends; they are slightly more negative and only small year-to-year differences. Inter-annual behaviour does differ, especially for relative humidity and especially in the period up to the early 1990s, where ERA-Interim is warmer and wetter generally, thus moderating the long-term trends in specific humidity, dew point temperature, and air temperature. Note that the ERA-Interim background field relative humidity shown in Fig. 1 also shows a decrease but to a lesser extent than the analysis fields (Fig. 9) which include ship data. Agreement is closest for air temperature in both trends and variability.

The decreasing relative-humidity trends over ocean are similar to the drying seen in HadISDH.land and ERA-Interim land relative humidity (Fig. 1); land linear trends are 0.03 %rh more negative, at -0.12 (-0.27 to -0.03) %rh per decade, over the same 1973–2018 period. The time series pattern is quite different though with marine relative humidity decreasing throughout the period around large variability and land relative humidity clearly decreasing from 2000. The greater sensitivity of relative humidity to observation errors, biases, and sampling issues makes the conclusion of long-term drying an uncertain one, but agreement with ERA-Interim adds some weight to this conclusion.

For the final HadISDH.marine.1.0.0.2018f product, the regional-average uncertainty is also computed and shown for the global average (70∘ S to 70∘ N) in Fig. 12. This includes the total observation uncertainty, which covers uncertainty components for instrument adjustment, height adjustment, measurement, climatology, and whole-number uncertainty (Table 2). In addition, the regional-average uncertainty includes the grid box sampling uncertainty and also a spatial-coverage uncertainty following the method applied for HadISDH.land (Willett et al., 2014). The coverage uncertainty essentially uses the variability between ERA-Interim full coverage compared to ERA-Interim with HadISDH.marine coverage to estimate uncertainty. To obtain uncertainty in the global average from the grid box uncertainties, correlation in time and space should be taken into account. It is not trivial to assess the true spatial and temporal correlation of the various uncertainty sources. In reality, although ships move around over space and time, implying some correlation, the contributing sources to each ∼500 km2 grid box monthly mean differ widely. Therefore, for this first-version product we assume no correlation between grid boxes in time or space and take the simple approach of the quadrature combination of uncertainty sources, noting that this is a lower limit on uncertainties.

The uncertainty in the global averages (Fig. 12) is larger than the equivalent time series for land (see Fig. 12 in Willett et al., 2014). The coverage uncertainty (accounting for observation gaps in space and time) is generally the largest source of uncertainty with the exception of relative humidity and dew point depression. For the latter two, the total observation uncertainty makes up the greatest contribution. In all cases the total observation uncertainty is larger at the beginning and especially the end of the records, where there are fewer or no metadata with which to apply bias adjustments. The contribution from sampling uncertainty (grid box spatial and temporal coverage) is generally very small except for relative humidity. This is as expected as the correlation decay distance of humidity should generally be larger over ocean than over land given the homogeneous surface altitude and composition. Overall, the magnitudes of the uncertainties are small relative to the magnitudes of long-term trends and variability in all variables except for relative humidity and dew point depression. This suggests that there is good confidence in changes in absolute measures of humidity over ocean (e.g. specific humidity) and also air temperature but lower confidence in changes in the relative humidity. The warming and moistening are further corroborated by strong theoretical reasoning based on laws of physics governing the expectation that specific humidity should have increased over the period of record given the warming of the oceans and atmosphere that has occurred (Hartmann et al., 2013). The uncertainty model makes many assumptions over correlation of uncertainty in space and time. It is likely that we have overestimated the uncertainty at the grid box scale by assuming complete correlation for height adjustment uncertainty, instrument adjustment uncertainty, and climatological uncertainty. Conversely, we have likely underestimated the uncertainty at the regional-average level by assuming no correlation. This is certainly an area for improvement in future versions.

Figure 13 shows the decadal linear trends for specific humidity, relative humidity, dew point temperature, and air temperature for HadISDH.marine.1.0.0.2018f. The completeness criterion for trend fitting is 70 %, more strict than for the climatologies (Fig. 7). This results in poorer spatial coverage especially in the Southern Hemisphere. Clearly, there are no data points outside 70∘ S to 70∘ N; hence the restriction of the global-average time series to this region is sensible. The tropical and Southern Hemisphere Pacific Ocean and Southern Hemisphere Atlantic Ocean have virtually no data coverage. Overall, the appearance of the trends shows good spatial consistency, with few grid boxes standing out as obviously erroneous. There has been no interpolation across grid boxes that would have smoothed out any outliers, and so the lack of these outlying grid boxes suggests that the data are of reasonable quality for this long-term analysis at least. Trends are as expected from the global-average time series – generally moistening and warming but becoming less saturated. The same is true over land (Willett et al., 2014).

The moistening shown in specific humidity and dew point temperature (Fig. 13a, b, e, and f) is widespread. The majority of grid boxes are considered to be statistically significant in that the 90th-percentile confidence interval around the trend magnitude is the same sign as the trend and does not encompass 0. The largest increases in specific humidity are in the lower latitudes, whereas the largest increases in dew point temperature are more spread out, with a tendency towards the extratropics and mid-latitudes. There are a few regions where there are clusters of grid boxes with drying trends. These are generally consistent between the specific humidity and dew point temperature, especially in the few cases where these negative trends are significant such as the central Pacific, the east coast of Brazil, the southern coast of Australia, and around New Zealand.

Marine air temperature shows widespread and significant warming, in agreement with HadNMAT2 (Kent et al., 2013). Very few of the grid boxes with a negative trend are significant. In some cases they are in similar locations to the drying trends seen in specific humidity and/or dew point temperature, e.g. the coast south of Australia around Tasmania, and the east coast of Brazil. The warming is stronger in the northern mid-latitudes, with the Baltic, Mediterranean, and Red seas showing particularly strong warming consistent with strongly increasing dew point temperature and specific humidity.

Whilst relative humidity is more sensitive to methodological choices and observational errors, the broad spatially coherent structures to the regions of increasing and decreasing saturation with broad-scale significance are very encouraging in terms of data quality. Furthermore, the drying trends tend to be around the mid-latitudes, while the increasing saturation trends are more around the tropics, as seen over land. We still urge caution in the use of marine relative humidity, but these results collectively suggest that decreasing saturation might be a real feature.