The primary goal of the data rescue efforts within UERRA was to improve spatial coverage of input data for future regional gridded and reanalysis climate products over the European domain. Our aim was not to develop single, long-term data series for particular stations, but rather improve the availability of sub-daily observations anywhere that may be underrepresented in the current observational data used for European reanalysis products. This involved, as a first step, identifying the basic station data used in current reanalysis products available at the European Centre for Medium Weather Forecasts (ECMWF) and other relevant databases that contain digitised observations.

To identify gaps in the available sub-daily climate record, we first conducted a visual examination of the data holdings of the International Surface Pressure Databank (ISPD, Cram et al., 2015) and the Koninklijk Nederlands Meteorologisch Instituut (KNMI) European Climate Assessment and Dataset (ECA&D: http://eca.knmi.nl/, last access: 12 January 2018). These databanks provide station lists, regularly updated datasets and online visualisation tools, making it relatively straightforward to identify the regions lacking in sub-daily data. We also examined the holdings of the national climate data systems of countries whose data may not yet be in a multi-national repository. In particular, we checked the data available from the national climate data management systems of countries that had not been included in previous regional data rescue projects, namely the Romanian Meteorological Administration (NMA-RO) and the national meteorological and hydrological services (NMHSs) of countries in the western Balkans, including Albania, Bosnia-Herzegovina, the Republic of Macedonia, Montenegro and the Republic of Serbia. With this data availability information, we identified the Mediterranean, eastern Europe and Scandinavia as three key sub-regions within the European sector as lacking in sub-daily data.

We then conducted an extensive examination of the data available for these regions within the Meteorological Archival and Retrieval System (MARS) at ECMWF. MARS is home to the primary data input for the current European reanalysis products available from ECMWF (e.g. Dahlgren et al., 2016), and so stations that are identified in data sources (see Sect. 2.2) but not present in MARS, or stations with low percentages of sub-daily data, are likely candidates for data recovery. Interrogating the MARS holdings is not as straightforward as ISPD or ECA&D due to the extremely large number of data sources stored in the system and the registrations required, which is why we conducted our search in this order.

We focussed our search on the three data-sparse sub-regions in the post-1957 period, to align with the temporal focus of the proposed UERRA regional reanalysis products and ECMWF historical reanalyses such as ERA-20C (https://www.ecmwf.int/en/research/climate-reanalysis/era-20c, last access: 22 July 2018). The variables of interest were several atmospheric and terrestrial essential climate variables (ECVs) as defined by the Global Climate Observing System (GCOS, World Meteorological Organization, 2015) that were identified as important for the development and verification of regional reanalyses: air temperature (TT), atmospheric pressure (sea level pressure, PP, and station level pressure, SP), wind speed (WS), wind direction (WD), relative humidity (RH), dew point temperature (DP), daily rainfall (RR), fresh snowfall (FS) and snow depth (SD).

The high percentage of stations with data for less than 60 % of the 1957–2010 period in MARS (Fig. 1) illustrates the lack of sub-daily observations in these sectors. Gaps are clear in the southern and eastern Mediterranean countries, Sweden, and Norway for the 1960s and 1970s (Table S1 in the Supplement), as well as across the Balkan region. The relatively dense spatial coverage of the stations with less than 60 % data coverage also suggests that sub-daily observations may have been taken at many places in these regions, but have not yet been made available in a standardised format.

As well as identifying gaps in the digitised sub-daily record available for Europe, we also needed to locate sources of undigitised sub-daily data. We undertook extensive consultation with NMHSs across the three identified regions of poor data coverage, in an attempt to identify and recover paper or scanned data sources suitable for digitisation. Priorities were given to data sources already available as scanned images, stations with data from the post-1957 period, and stations where the selected ECVs were recorded (see Sect. 2.1). Recovered precipitation observations from NMA-RO were digitised internally, and then provided to us in digitised quality-controlled format, using a similar quality control format to that used in this study (see Sect. 3.2). Discussion with the Norwegian and Swedish NMHSs uncovered data for these countries that had been digitised, but were not yet provided to international data repositories. Similarly, the Catalan Meteorological Service (MeteoCat), which has an open data policy, allowed their digitised data for the recent 1998–2015 period to be transferred to relevant global repositories through our effort. Data sharing was organised between these regions and ECMWF without the need for observations to be transcribed from paper format and will therefore not be discussed further here. Political and financial difficulties prevented many other countries we contacted, particularly in northern Africa and the Balkans regions, from providing original data sources to us for digitisation.

Original data sources were provided in scanned format by Deutscher Wetterdienst (the German Meteorological Office, DWD), the Slovenian Environmental Agency (SEA), and Agencia Estatal de Meteorología (the Spanish Meteorological Service, AEMET), via MeteoCat. Close consultation with these NMHSs enabled us to identify valuable and previously undigitised data sources. From these sources, stations with minimal data available in MARS were selected for digitisation.

The World Meteorological Organization (WMO) Mediterranean data rescue initiative MEDARE and the precursor project to UERRA, the European Reanalysis and Observations for Monitoring project (EURO4M, http://www.euro4m.eu/, last access: 12 January 2018), located key records of data for the Middle Eastern, Balkan and southern Mediterranean regions from the Serbian NMHS online climatological scanned repository (http://www.hidmet.gov.rs/ciril/meteorologija/klimatologija_godisnjaci.php, last access: 4 June 2018), the United States of America's National Oceanic and Atmospheric Administration/National Climatic Data Center (NOAA/NCDC) Climate Data Modernization Project (CDMP: http://library.noaa.gov/Collections/Digital-Documents/Foreign-Climate-Data-Home, last access: 8 August 2018), the British Atmospheric Data Centre (BADC, http://badc.nerc.ac.uk/browse/badc/corral/images/metobs, last access: 8 August 2018), and other national meteorological services (see Brunet et al., 2014a, b for details). Daily maximum and minimum temperature, precipitation, and sub-daily atmospheric air pressure observations from some of these sources were digitised under the auspices of EURO4M and MEDARE, but many other observations were unable to be transcribed due to project constraints. UERRA therefore provides a valuable opportunity to rescue the previously undigitised values from these sources (Brunet et al., 2014b).

Table 1 provides detail of the data sources identified for digitisation, while Fig. 2 shows several examples of the data sources used. All of the variables included in each source are listed in Table 1, although not all were digitised under the auspices of UERRA. The majority of data sources from CDMP are secondary, meaning that they are collations or summaries of observations that have been prepared in a central location. Unfortunately, secondary data sources are more prone to transcription errors than original series, as they have been transferred from the original readings. Many were handwritten, although a small subset was typed.

Once data sources had been identified and catalogued, a group of 11 digitisers were employed for 15 h a week over a 2-year period to digitise the data. The digitisation team was made up of undergraduate and postgraduate geography students from the Universitat Rovira i Virgili (URV), who all had some knowledge of meteorological variables and European climate. The digitisers worked on desktop computers in a computer lab, with large screens and standard keyboards. They were also given the option of working from home on their personal laptops.

The digitisers received initial training sessions, online instructions and monthly in-person meetings to discuss issues and introduce new digitisation tasks. Digitisation was done using a “key as you see” method, meaning that the digitisers typed the values they could read in the data images, rather than using any coding system. This follows standard best practice outlined by the WMO (2016). Clear, unambiguous errors in the data sources were generally retained by the digitisers and recorded in station metadata files, which were later used when quality controlling the data (see Sect. 3). If a digitiser could not read a value due to poor handwriting or scanning issues, they represented it by a value of -88.8, while missing values were set to -99.9.

Budget constraints made it unfeasible to use double-keying, a suggested method of improving digitised data quality where the same data are transcribed twice (Brönnimann et al., 2006; World Meteorological Organization, 2016). We tested optical character recognition (OCR) and speech recognition technologies, but the diverse nature of each task and the time and cost associated with training the software to each data source made these options unfeasible. However, the digitisers were trained in self-assessment techniques aimed at reducing data errors. Digitisers were asked to carefully cross-check their values with the original source values for the 10th, 20th and 30th day of each month to make sure that no days had been skipped or repeated. Days with missing data were recorded in metadata files, along with any other variations in the data source, such as repeated pages in the scanned file or temporary changes in the table structure. Where data sources included monthly totals and summaries, digitisers were also instructed to calculate these values from their daily transcribed data, to check accuracy.

The data sources were in a number of different formats. The two main formats were 1 month (or day) to a page for a single station, and 1 day to a page for a network of stations. Depending on the source structure, each digitiser was in charge of digitising values from a station (e.g. Egyptian and Moroccan sources, Fig. 2a and b), a time period (e.g. Slovenia, Fig. 2c) or a variable (e.g. Lebanon, Fig. 2d). English and Catalan translations of the relevant column and row headings were provided to the digitisers for each source, as well as the various wind strength scales (see Sect. 2.4).

In several cases, not all of the data on a sheet were required to be digitised, as they had already been transcribed as part of EURO4M and MEDARE. To help digitisers with the complex layout of the source images, templates were developed in Microsoft Excel for some sources that were as close as possible to the format of the original data source (see Fig. 3 for several examples). Borders and shading within the files were used to help the digitiser keep track of their work, and date columns were pre-filled with the correct dates to reduce the occurrence of errors associated with leap years. While the development of templates was not always possible due to time constraints, templates were used for all sources with very high-resolution data (e.g. observations every hour, see Table 1).

The digitisers were required to upload their data to a central server every 15 days, include a count of the number of values digitised and include an up-to-date copy of the data transcribed. This method ensured that the digitisers were making progress, the data were being regularly backed up and the digitised observations could be regularly checked (see Sect. 3).

While visual quality control and assessment were applied to the data in their original units, the data were also converted to standard units, to be used in widespread meteorological products and statistical quality control procedures (Table 2). Data sources and available metadata were examined closely to ensure the conversions were as accurate as possible, and any changes to units within the same source were captured. Many atmospheric pressure observations in particular needed to be converted from millimetres of mercury to hectopascals, and station level pressure data reduced to sea level pressure for quality control testing. This step involved a detailed examination of the data sources to identify station height information and any instrument movements that may have occurred. In most cases, only the station height information could be located, but any changes identified were recorded in the coordinates accompanying the final dataset.

A selection of values uploaded by digitisers were systematically compared to the original source images by postgraduate researchers and other digitisers at the Centre for Climate Change at URV familiar with the sources. The aims of these initial visual cross-checks was to provide timely feedback to the digitisers if common digitisation errors were occurring, identify subtle errors in the order of the data that may not be picked up in statistical procedures and also make a preliminary assessment of the quality of the data from each particular source (Table 1). Additionally, regular reporting of data completed helped us identify any digitisers who were having trouble with their tasks and needed extra assistance.

For every fourth year of data, 2 or 3 days of observations were selected at three monthly intervals for visual cross-checking with the original source. This was completed for data from all sources. Additional ad hoc checks were made if a known issue existed in the data source, e.g. if the period covered by the data source contained a leap year, or the source pages were known to be out of order. Although these checks only covered a small percentage of the total digitised data, we felt it was sufficient to identify the general quality of work done by individual digitisers and for each source.

In more than 60 % of stations tested, only a small number (less than 5 %) of the checked values required correction. Visual cross-checking of data from stations with a larger number of errors identified the occasional skipped day or duplicated value, which meant that a large percentage of observations needed to be shifted by one time step. The majority of these errors were found in data for Egypt and Algeria, from sources that had already been flagged as difficult to read and containing date order errors. In two cases, digitisers were asked to repeat their work.

After the basic visual cross-checks, the digitised data were subjected to a range of statistical quality control tests. Due to the highly variable nature of the different data sources, and their disparate geographical spread, data from each station were examined individually. Data were also examined in their original temporal resolution, and not converted to daily averages, as averaging the sub-daily values would make it difficult to identify the erroneous value. Statistical quality control was conducted using a semi-automatic quality control (SAQC) procedure (Universitat Rovira i Virgili, 2014). The SAQC method was largely adapted from existing automatic quality control procedures developed for sub-daily data at a global scale (e.g. Dunn et al., 2012; Durre et al., 2010), but was modified for our dataset to enable more manual examination of the resultant flags. Full details of the procedure, the relevant software and instructions for use are available from A.Q.C. Software menu at http://www.c3.urv.cat/softdata.php (last access: 8 August 2018).

SAQC comprised of three separate programs that can be applied to the data at their original time resolution in text file format: one examining temperature, wind, relative humidity and dewpoint observations; another assessing sea level pressure data; and a final check on sub-daily rainfall data, daily snow depth and snow fall data. The tests applied within SAQC (Table 3) can be largely grouped into four groups depending on the degree of QC applied (Aguilar et al., 2003):

Gross errors tests. These are QC tests that detect and flag obviously erroneous values (date order check, date errors, unrealistic values, data repetitions and non-numeric value tests).

Each program produced a list of values flagged by each test at each station. The combined key results were then manually cross-referenced against the original source data, and corrected or removed from the quality-controlled version of the dataset. The removal or correction of each value was recorded using a flag system, to clearly document the nature of the identified errors and results (Table 4). An example of the air temperature evolution in Port Said (Egypt) taken at 08:00 and 14:00 local time for the short period 1939–1940 and resultant QC flags is shown in Fig. 5, highlighting various types of errors, outliers and extreme values over a short time period.

In the initial testing of the SAQC procedure, the tests for duplicate values, monthly mean of absolute increments and unusual distribution of values tests were found to be overly sensitive, resulting in many valid observations being flagged for assessment. Many of the legitimate errors identified by these tests were also found by others, so the thresholds on these tests were relaxed to make the task of checking flagged values more manageable.

The final QC procedure consisted of subjecting data from neighbouring stations to spatial quality control tests, as well as rerunning several individual station checks in a fully automated way as a second-round check for gross errors that may have slipped through SAQC. Only data that had been checked by visual means and SAQC were subjected to this procedure and as with SAQC, the data were examined in their original temporal format to avoid removing valid data. This QC process (Hadley quality control, or HQC) was conducted using an adapted version of the procedure used in the development of the UK Met Office Hadley Centre Global Sub-Daily Station Observations dataset (HadISD v2.0.1.2016p; Dunn et al., 2012, 2016). Due to time constraints, only data digitised as part of this project were used in the spatial quality assessment, although future work could make use of the existing HadISD dataset as a reference network.

Automatically running HQC with the standard thresholds used in the development of the global HadISD dataset led to a large number of false positive flags being identified (Fig. 6), as the rescued dataset had low spatial coverage and included observations taken at inconsistent times, often converted from units with coarse resolution. To reduce the number of false positive flags and increase the number of stations that could be checked, some of the HadISD tests were adapted (Table S2). The minimum number of neighbouring stations required for HQC testing was reduced from 10 to 5, and the percentage of non-missing observations per month allowed was reduced from 75 % to 66 %. Tests that looked for streaks of identical values, or non-uniform distributions in the frequency of values, were also slackened to account for the fact that many observations were converted from different units.

Data from each country were then split into networks according to their correlation, spatial distance, observing times, overlapping observing periods and variables observed. Six appropriate networks were identified (Table 5), but it was not possible to include all stations, periods, variables and observing times. The heterogeneous characteristics of the dataset, the high spatial separation and irregular distribution of the stations, and the inconsistent coverage of the variables included in the dataset meant that only about 4.3 million observations (over 48 % of the total dataset) could be subjected to HQC.

For example, it was not possible to apply HQC to data from Cyprus, Lebanon and Spain due to the low number of stations in each country and the large distance between the stations of neighbouring countries. We were also unable to automatically analyse fresh snow and snow depth, precipitation, or relative humidity data, as the HadISD QC does not assess these variables as raw input. Moreover, several stations (such as those in Germany and Slovenia, the central Europe network in Table 5) provided hourly data, but there were not enough neighbouring stations with sufficiently high temporal resolution to allow for more than a subset of observing times per day to be checked.

A total of 8.8 million observations were digitised from 127 stations in 15 countries (Tables 6 and S3). Long records (>30 years) of many variables were successfully recovered from stations in Egypt, Tunisia and Algeria, although only the Egyptian stations provided observations more than once a day (Fig. 7). Shorter but more widespread observations were rescued across Morocco, Turkey and the Balkans region, while the snowfall observations in Germany only covered the west of the country.

The largest number of observations (more than 28 %) came from Slovenia (Fig. 8a); even though we only had data for three stations in Slovenia, the observations were hourly, included nine variables and covered more than 20 years. Around 15 % of the rescued observations came from Egypt, and almost 12 % from Turkey. Both of these countries have a large number of stations in the recovered network, and a variety of variables over a long period of time (Fig. 7).

More than 21 % (1.8 million) of the rescued observations were sub-daily temperature measurements, with wind speed and direction measurements totalling over 17 % (Fig. 8b). There were around 20 000 more wind direction observations than wind speed observations; this is because very early Tunisian and Egyptian wind speed observations were qualitative (e.g. light, moderate) and were not digitised. Relative humidity data made up around 16 % of the rescued dataset, while sea level pressure and station level pressure contributed a similar amount at just over 15 % (around 1.4 million values). Over 160 000 fresh snow and 160 000 snow depth values (more than 3.5 % of the full dataset combined) were also recovered from Germany and Slovenia from as early as the 1950s, representing a significant increase in snow observations across the region.

Due to the temporal coverage of the Slovenian data (1950–1978), as well as the dedicated focus of the UERRA project on post-1957 observations, the mid-20th century was the most well represented period in the rescued dataset (Fig. 8c). Almost 60 % of the dataset covered the 20 years from 1950 to 1969. Observations from Cyprus and northern Africa provided data from the late 19th century, and records from Serbia were recovered up to 2012.

Finally, the most common observing times for the variables rescued were 07:00, 14:00 and 21:00, reflecting standard observing practices over the European region in the 20th century. Tunisian observations were only available for 07:00, and for many other countries where observations were only available once a day in the early part of the record, these observations were also inevitably in the morning. Two German stations included a small number of half-hourly observations (Fig. 8d).

All rescued sub-daily data were subjected to quality control routines to identify erroneous values or chains of values in the time series (Sect. 3). A total of 3.2 % of observations, around 268 000, were flagged as suspicious for the whole dataset using SAQC (Fig. 9).

Flagging correct values (false positives) is a common QC issue, and manual examination ensured that these important observations – often of extreme events – are retained for future studies. The majority of the values flagged (1.5 % of the total number of values) were corrected after manual examination, with just over 1 % of the total number of observations removed from the quality-controlled version of the dataset due to errors in the source image or issues with the readability of the original values. This includes observations recorded as -88.8 by digitisers (hard to read, see Sect. 2.3). Over 27 000, or 0.3 % of the total number of observations, were flagged but then found to be correct after examination.

Despite being among the countries with the smallest number of observations, the largest percentages of flagged values found were for Bosnia-Herzegovina and the Czech Republic (∼8 % of the total number of data digitised, Fig. 10a). For Bosnia-Herzegovina a large section of observations from one station was given a flag of fl11 and removed due to an extensive digitiser error that could not be reconciled. A digitisation error in the Czech Republic observations was able to be corrected by shifting data by 1 day, resulting in a large number of fl12 flags (corrected based on original source). The hand-written nature of the Czech data, together with the absence of data templates (only used in Slovenian, Spanish and German data sources) may go some way to explaining the large number of flagged values among both countries. The countries with the largest number of observations (Egypt and Slovenia) had about 3 % of their observations corrected or verified and less than 2 % removed under the SAQC procedure.

A similar amount of flagged values were proportionally found in all rescued observations distributed by variables, except for precipitation (RR, Fig. 10b), which was only available for Slovenian stations. The high number of precipitation flags is due to two factors. Firstly, several digitisers inadvertently recorded zero rainfall values as missing, or missing rainfall as zero. The format of the Slovenian data sources changed over the period, with some years having hourly rainfall data and others only providing observations 3 or 4 times a day. Reporting no rainfall as missing data could significantly affect any future analysis of rainfall frequency using these data, and so these values were corrected, resulting in a number of fl12 (corrected based on original source) flags. Secondly, during the latter part of the Slovenian record, some daily rainfall totals were calculated inconsistently, using a midnight-to-midnight sum occasionally rather than a 07:00–07:00 total. The 6-hourly observations from the same stations were quality-controlled based on these totals, but the daily rainfall totals calculated in this way were removed from the final version of the dataset, to ensure consistency, and given a flag of fl15 (removed due to source error).

SAQC flags distributed by decade show a similar pattern to the distribution of observations, with a peak in the mid-20th century (Fig. 10c). The higher number of fl17 flags (observations set to missing as no value could be found in the source image) during the 1940s may reflect data issues during the Second World War, particularly for Egypt and Algeria, where some original source files were ordered incorrectly. This resulted in a number of values being ascribed to the wrong date. Flagged values were relatively evenly distributed across observation times (Fig. 10d), although the lower absolute numbers of half-hourly observations made for a higher proportion of flagged observations during these times (compare Fig. 8d and 10d).

In total about 64 000 values were flagged and subsequently removed by HQC, around 0.7 % of the total dataset. Temperature was the variable with the smallest number of flagged values overall by HQC, with the exception of the northern African network, where data source resolution and the high number of missing values caused HQC to flag and remove extra values (Fig. 11). The variable with the highest proportion of flagged values in the northern African network was sea level pressure.

Given the automatic nature of the HQC tests, all values flagged by this step were removed from the final version of the dataset and given a flag of fl36. Values that were subjected to HQC were therefore marked with an additional flag (a prefix of 3), to clearly identify the level of testing applied to each individual observation (see Table 5 and Fig. 12). For example, observations which were corrected or verified in the SAQC round of testing and given an initial flag of fl12 or fl14 but passed the HQC procedure had a final flag of fl32 or fl34, ensuring that information from both rounds of QC were retained.

While the HQC tests were unable to be applied to all of the observations, these results are similar to the findings of the HadISD spatial QC analyses (Dunn et al., 2012). Around 3.9 %, or about 330 000 observations, were flagged by both QC procedures (Fig. 12). A total of 2.1 % of the data were removed as a result of SAQC and HQC testing, with 1.5 % corrected during the SAQC process. Only 0.3 % were flagged but later verified during SAQC, although this includes many legitimate extreme events that are crucial for calibrating and verifying the tails of atmospheric behaviour which can have the largest societal impact. These percentages of flagged values are similar to those identified by Brönnimann et al. (2006), who found transcription error rates of 0.2 % to 3 % for hourly temperature and upper air observations.

In the final data check, a small conversion problem was detected with the atmospheric pressure at two Slovenian stations (around 318 000 values). The vast majority of these observations passed both SAQC and HQC, with large errors identified and flagged appropriately. However, these observations were marked with a prefix of “4” rather than “1” (subjected to SAQC) or “3” (subjected to SAQC and HQC) in the final dataset, to signify that additional QC may be required by future users.

Incidental errors throughout the digitisation process, namely digitisers keying the same data twice, gave us an additional opportunity to examine the quality of several data sources. In particular, these opportunistic analyses allowed us to identify the likely percentage of errors that would be identified using a double keying technique.