A database of databases for Common Era paleoclimate applications

We present a merged database of five curated databases (DoD2k version 2.0.0) developed for Common Era (1–2000 A.D.) paleoclimate research. A toolkit to create the database is also provided and leverages codebases developed by individual database developers and the paleoclimate data informatics communities over the past decade. It includes Python notebooks for (1) loading each database from its public repository, using a common, compact set of terms for metadata and data management; (2) their merger using that common set of dictionary terms; (3) a multistage algorithm to identify candidate duplicates; (4) an operator-supervised, semiautomated decisionmaking procedure, checking against a common set of metadata and comparison metrics; and (5) creation and checking of the finalized, duplicate-free database across 22 dictionary terms. Each of the curated databases, which arise from individual, community and PAGES (Past Global Changes) 2k working groups, represent a range of development approaches, from single archive, single observation datasets with a range of reported environmental responses, to multiarchive-multiobservation collections which target a specific climate response.

The resulting DoD2k spans 13 archive types, 37 data types, and 4781 records within the Common Era. We illustrate the value of the DoD2k with two applications. In the first, we extract the moisture (M) and temperature (T) sensitive subset of records and perform an empirical orthogonal function (EOF) analysis on the resulting multi-archive, multi-observation MT-sensitive dataset. In the second, we show that calcite speleothem oxygen isotopic composition is consistent with that simulated using simple models of this proxy system. DoD2k may also be useful for paleoclimatic detection and attribution analysis using proxy system modeling, data assimilation, and deep learning for the development and testing of improved proxy system models. The database can be accessed at https://doi.org/10.25921/sptp-g618 (Evans et al., 2025). The toolkit can be accessed at https://doi.org/10.5281/zenodo.15676255 (Luecke et al., 2026).

1 Introduction

The climate is changing, forced primarily by human-caused increases in greenhouse gas concentrations, aerosols and land use change, toward a warmer and more moisture-inequitable state, in which extreme events are more likely, and more extreme, than observed during the 20th century (Arias et al., 2021). Superimposed on that are other causes of forced climate variation and change, for instance, arising from volcanic activity, solar and orbital variations (Huybers and Curry, 2006; Miller et al., 2012; McGregor et al., 2015; Abram et al., 2016; Gebbie and Huybers, 2019). In addition, there is the tendency of the climate system, broadly defined as the coupled ocean, atmosphere, land surface, land and sea ice, biota, and solid earth, to vary on a broad range of timescales under stochastic forcing (Hasselmann, 1976). How then, to define both the spatial imprint and the amplitude of the climate change that arises from such forcings, and distinguish it from the human-driven forcings? The answers are important: first, for defining the equilibrium and transient climate change in response to a unit of forcing, over time and in different parts of the world (Forster et al., 2021); second, for detecting and projecting the impacts of both anthropogenic and natural climate forcing over past and future decades and centuries (Fox-Kemper et al., 2021; Marvel et al., 2019b, a). For such goals we need realistically forced paleoclimate simulations and observations (Neukom et al., 2019a).

The development of the observational target for such work is the focus of the present contribution, in particular for the so-called Common Era (1–2000 CE), for which observations from paleoclimatic archives are most dense and diverse, and permit an approximate 10-fold increase in the time interval of study relative to the historical record. More specifically, we would desire the most dense and random sampling in space and time, of all possible observations, imprinted with a diverse set of climatic information, and resolving timescales of variation from subannual to multicentennial with similar observational temporal resolution, and with well-characterized chronological uncertainty. The natural starting point for such an effort would be public repositories of individual paleoenvironmental datasets and databases, such as at the National Center for Environmental Information (https://www.ncei.noaa.gov/products/paleoclimatology, last access: 15 January 2026) and PANGAEA (https://pangaea.de, last access: 15 January 2026). However, this is impractical for multiple reasons, including some nonuniformity in dataset submissions and metadata, changes in repository submission templates and requirements over time, and the presence of multiple versions of datasets in repositories whose prime directive is preservation and availability (Anderson et al., 2019).

An alternate foundation for the development of such a database is in existing databases, many of them compiled by years-long efforts by PAGES (Past Global Changes; https://pastglobalchanges.org/, last access: 15 January 2026) Working Groups. PAGES databases are the result of leveraging community-level specialist expertise that is difficult for any single research group to assemble or maintain. The work of many individuals in multiple groups has enabled the development of publicly available observational datasets, the metadata that describes them, and most recently, the open semantic formalisms (Emile-Geay and Eshleman, 2013) and codebases (McKay and Emile-Geay, 2016) that enable their re-use. However, each such database, although rich in metadata, metadata uniformity, error checking and quality control, is generally assembled for a specific purpose. For example, the PAGES 2k Consortium (PAGES 2K Consortium, 2013; Emile-Geay et al., 2017) originally planned development of a multiarchive database (wood, coral, ice, documents, lake and marine sediments) of many different temperature-sensitive observations in these archives for the purpose of global mean and spatially resolved surface temperature reconstructions (Neukom et al., 2019a, b). The SISAL Working Group (Kaushal et al., 2024a) developed a single (speleothem) archive of multiple observations (e.g. δ¹⁸O, $\mathrm{Mg}/\mathrm{Ca}$) made in that particular archive. The Iso2k Working Group (Konecky et al., 2020a) produced a multiarchive (marine sediment, lake sediment, marine carbonate, speleothem carbonate, wood, ice) database of solely δ¹⁸O and δD observations in those archives, agnostic of climatic interpretation. For facilitating the repurposing of these databases for other scientific goals, such as the reconstruction of hydroclimatic variability (Falster et al., 2023) and the detection and attribution of climate change in both moisture and temperature via paleoclimatic data modeling (Franke et al., 2022a), we might need to combine multiple existing databases.

There are multiple challenges to creating a unified database of databases for Common Era paleoclimate applications. These include differences in the metadata and terminology for describing datasets across databases but within even the same proxy observation types and biogeochemical archival materials; differences between databases of the required metadata, sampling resolution, age model development, time resolution, level of replication, descriptions of observational uncertainty, and interpretational notes; and the problem of duplicate detection across combined databases (Anderson et al., 2019; Tardif et al., 2019; Steiger et al., 2022). Unfortunately, differences in metadata and terminology for defining properties across different curated databases, as well as differences in database terminology, structure, management, make merging databases and cleaning them for duplicates difficult. For instance, PAGES2K (Emile-Geay et al., 2017) has 173 dictionary terms, and as it happens, SISALv3 (Kaushal et al., 2024a) has 173 unique dictionary terms linking its 21 constituent csv files into a database. However, these are not the same 173 dictionary terms as for PAGES2k. Although there is common overlap in metadata, such as site identification name, they have different keys ('paleoData_TSid' in PAGES2K, 'site_id' in SISALv3).

Here we describe a unified Database of Common Era paleoclimate Databases (hereinafter, DoD2k) that addresses these challenges. We also provide a flexible and open framework, hereinafter called dt2k, implemented as Python functions, scripts and Jupyter notebooks, in which databases to be merged are loaded along with a standard set of metadata by which to merge existing databases may be specified (McKay and Emile-Geay, 2016), and in which duplicates across merged datasets may be identified and removed. The framework is extensible and can incorporate new databases or updates to existing ones. It provides methodical and comprehensive testing for duplicate records and can be easily attached to paleoclimatic analysis and reconstruction toolsets (Zhu et al., 2023, 2024).

The existing Common Era paleoclimate datasets to be merged, and the process by which they are merged into DoD2k, are described in Sect. 2. DoD2k is described in Sect. 3. Some applications using DoD2k are illustrated in Sect. 4. An outlook for future development is in Sect. 5 conclusions are in Sect. 8.

2 Data, Workflow and Codebase

An overview of the workflow for creating DoD2k is shown in Fig. 1. DoD2k the product, as well as dt2k, the codebase for its production are supplied in a GitHub repository (https://www.github.com/lluecke/dod2k, last access: 15 January 2026). The codebase is written in Python 3. We provide a virtual environment (dod2k-env) and dt2k in a series of Jupyter notebooks and python scripts.

Figure 1 Schematic overview of the DoD2k workflow, following Sect. 2. From upper left: starting with the (2.1) databases to be aggregated, we (2.2) load each dataset as a compact common subset of the metadata and data using a standard set of dictionary terms, if necessary translating from the original terms, concatenating the data. We then (2.3) detect duplicates using an automated scheme before (2.4) performing an operator-supervised, journaled duplicate management decision on each candidate set (see text). Step (2.5) is the removal of the duplicates, with a second check for copies than might appear accross more than one database (not shown in figure), before (2.6) the DoD2k is written and commented, for subsequent review, reproducibility of processing and decisionmaking, and traceability back to original databases and their entries (Bush et al., 2020).

A thorough documentation of the toolkit, including a series of tutorials (https://lluecke.github.io/dod2k/, last access: 15 January 2026) guides the user in the loading and visualizing of DoD2k and the application of dt2k to produce DoD2k (this section). Separately the user can examine the steps of loading and merging databases and duplicate detection and removal for other applications. Thus, we introduce here not only DoD2k, but also the opportunity for users to modify dt2k based on their specific requirements, including but not limited to, add data, add variables, and make their own expert decisions regarding the potential duplicate candidates.

2.1 Data

We assemble five curated databases of Common Era paleoclimate records from which we obtain DoD2k (Fig. 1). These are:

• Breitenmoser et al. (2014) (hereinafter, fe23): restandardization of tree-ring width (TRW) chronologies for comparison with their 20th century simulation using the VS-Lite data model (Tolwinski-Ward et al., 2011), with data available within the interval 850–2000 CE and with climatic interpretations re-estimated as published in a subsequent study (Franke et al., 2022a), served from https://www.ncei.noaa.gov/metadata/geoportal/rest/metadata/item/noaa-model-36773/html (last access: 15 January 2026);

• Emile-Geay et al. (2017): PAGES2k v2.0.0 (hereinafter, p2k): multiproxy, multiarchive compilation of temperature-sensitive proxies for the Common Era, updated and served from https://lipdverse.org/Pages2kTemperature/2_2_0/ (last access: 15 January 2026);

• Konecky et al. (2020a): Iso2k v1.1.2 (hereinafter, iso2k): multiarchive compilation of oxygen and deuterium isotopic records extending through the Common Era and into previous periods, updated and served from https://lipdverse.org/iso2k/current_version/ (last access: 15 January 2026);

• Walter et al. (2023): CoralHydro2k v1.0.1 (hereinafter, ch2k): single archive, multiproxy compilation of records from coral carbonates, within the Common Era, updated and served from https://lipdverse.org/iso2k/1_1_2/ (last access: 15 January 2026);

• Kaushal et al. (2024a, b): SISAL2k v3 (hereinafter, sisal): single archive, multiproxy compilation of cave carbonate records, extending from the present through the Common Era and into previous periods, served from https://lipdverse.org/CoralHydro2k/1_0_1/ (last access: 15 January 2026).

2.2 Load and merge databases

To assemble a database of databases, we identified a set of 19 metadata fields (Table 1) which satisfy the following criteria across all the individual databases. Apart from a few fields specific to DoD2k, the nomenclature of the fields was largely adapted from Emile-Geay et al. (2017) and the PAGES2k v2.0.0 temperature-sensitive database, and then expanded through the adoption of the PAGES2k v2.2.0 update and additional reviewer-suggested fields (this study). Criteria were that the metadata is commonly used within the community, and the majority of the original databases have non-missing entries within the field. For a description of how the individual metadata parameters were collected, we refer back to the original databases and their development by teams of specialist researchers. Note, however, that this set of fields is subjective and by no means exhaustive; if desired by the community or by an individual user, may be expanded with modification of the load notebooks provided. For example, the p2k v2.0.0 database had 173 metadata fields. However, half of these fields were missing for 85% of entries, making it more difficult for users to identify and extract relevant metadata across all concatenated databases. We have selected fields that are a balance between essential data such as location, “least common metadata” available across most individual databases, and metadata such interpretation_variable. Three additional fields were added in response to reviewer comments to an earlier version of the load notebooks. The resulting 21 fields are illustrated in Fig. 1 and listed in Table 1; a 22nd field, duplicateDetails, is added following the duplicate detection, decision and removal process (Sect. 2.3).

Table 1 Metadata and data parameters of DoD2k, by common compact dictionary key. Note that the Key duplicateDetails is added as a result of the duplicate screening process (Sect. 2.3).

Emile-Geay et al., 2017(Breitenmoser et al., 2014; Franke et al., 2022a)(Kaushal et al., 2024a)(Walter et al., 2023)(Konecky et al., 2020b)

Table 2 Notebooks for the creation of DoD2k, their input data sources and output files. INITIALS: Operator's initials as specified in the notebook; YY-MM-DD: datestamp of operator decisions.

The load notebooks read the five original databases (ch2k, fe23, iso2k, p2k and sisal), extract a number of shared variables, and concatenate the databases. Each notebook (Table 2) follows the structure:

• set up environment, load source data retrieved by user from the source public repository.

• potentially process data according to database provider (may use code supplied by authors of the original database) to obtain a pandas dataframe.

• identify and extract the relevant variables.

• convert metadata and data to the correct format, using standard terminology where applicable (following the LiPDverse controlled vocabulary).

• save output as a “compact” pandas dataframe with standardized metadata.

The output of each load notebook is a standardized “compact” set of 21 metadata and data fields (Table 1). Output is saved in pickle (.pkl) format, as well as a series of comma separated value files. A unique datasetId is pulled from the identifier used in the curated dataset of components. Each additional metadata field contains either information from the original dataset as published, or has been added for the purpose of improving data identification. In the case of the former, some information has been homogenized to a standard set of values across the databases, e.g. in case of the archiveType: “Wood” to identify all tree-ring type proxies, or for interpretation_variable: “T” to identify all temperature sensitive proxies. Thus, some of the information may slightly differ from the original authors in terms of nomenclature. Our nomenclature closely adheres to the LiPDverse controlled vocabulary (https://lipdverse.org/vocabulary/, last access: 15 January 2026) where applicable, to promote a more standardized language within the paleoclimate community. This allows filtering of the database for individual values, and thus easy access to e.g. a database for an individual archive type. We have also transformed chronological assignments such that they are all provided in units of years CE, and we have restricted the data to the Common Era, 1–2000 CE.

The resulting homogenized original databases are subsequently assembled into DoD2k, using the notebook merge_databases.ipynb, after which the merged database is ready for duplicate screening.

2.3 Duplicate detection

The duplicate screening process consists of three separate steps (Fig. 1, steps 2.3–2.5), each implemented as a separate notebook. This process uses the output of any of the load notebooks; here we describe its application to the merged database of databases.

The first step, duplicate detection (dup_detection.ipynb), identifies the potential duplicate candidates. Duplicate detection begins with a simple threshold correlation coefficient test between any two records with paleoData_values x and y. We choose $\text{corr}(x,y)>\mathrm{0.98}$, which we consider sufficient to find exact duplicates, but also allows detection of records which, for example, might differ by only a few points at the beginning or end of the observed time interval, have a limited number of missing values, or contain a different number of retained significant digits. However, this criterion is not sufficient on its own (as discovered in analysis of the Steiger et al. (2022) database) to identify duplicates that might arise from standardization choices, compositing, truncation, and metadata differences that might differ across databases. We therefore add multiple additional diagnostics, metadata comparison, and operator screening of candidates. Here, any two records with paleoData_values x and y, and their associated z-scores z_x and z_y, are scanned according to the following necessary criteria:

1. data type: archive type (archiveType) and proxy type (paleoData_proxy) must be identical,

2. site: there must be an overlap in the site name (geo_siteName), e.g. at least one shared expression,

3. location: distance between the point coordinates (geo_meanLat and geo_meanLon) ≤ 8 km,

4. correlation: $\left[\text{corr}(x,y)>\mathrm{0.9}\vee \text{corr}(z_x,z_y)>\mathrm{0.9}\right]\wedge \left[\text{RMSE}(x,y)<\mathrm{0.1}\vee \text{RMSE}(z_x,z_y)<\mathrm{0.1}\right]$

5. overlap: time intersect (year) at least ≥ 10 points, unless one of the records is overall shorter than that, in which case the other criteria are sufficient,

6. URL: the data identifier (originalDataURL) are identical if both candidates originate from the same database (originalDatabase), otherwise the other criteria are sufficient.

The so-detected candidate pairs are flagged, and their summary figures as well as a list of the candidate pairs are saved. In addition to the 21 fields previously described, the assembled and duplicate-screened database DoD2k contains the field duplicateDetails (Table 1). This field is populated during the duplicate screening process and includes information on the detected duplicates and any choices made by the user, such as the selection of one record or the other of the candidates, and whether a composite record has been created (see below).

2.4 Duplicate decision

The second step, the decision process implemented in the notebook dup_decision.ipynb, uses the output generated in the previous step to perform a decision process on each candidate pair. In order to make this process more operator-friendly, we have implemented a mixture of automated and user-operated decisions. For candidate pairs which are evidently true duplicates, no user input is necessary. Instead, a record is removed, based on a user-specified database hierarchy (see next paragraph). The evidently true duplicate records satisfy the following criteria:

• 7.

  geo_meanLat and geo_meanLon agree up to a decimal place,

• 8.

  geo_meanElev agrees up to 1 unit,

• 9.

  archiveType, paleoData_proxy, originalDataURL and geo_siteName are identical.

These user-specific criteria were empirically determined during the decision process and are thus tailored to our purposes. They were based on a number of true duplicates, where the geographical location differed slightly due to different authors' precision in the specification of coordinates.

We also specify a hierarchy for adoption of duplicate records from databases, with the assumption that newer databases were improvements over older ones, i.e. p2k > sisal > ch2k > iso2k > fe23. In addition, no user input is needed either for cases in which one of the candidates is evidently a recollection/update of the other record. Here, the geo_siteName is scanned for the keywords “update” and “recollection”, and to determine if the location criterion (see #3 in the list above) is satisfied. We also include the capacity to automatically choose one candidate over another in a consistent way for series whose metadata is identical, but the records are nonidentical. In DoD2k, this is used to automatically select fe23 AR-standardized tree-ring width records over the same p2k records. This was done because in fe23, n≥8 replicate observations are required and the 1901–1970 interval is complete to produce a chronology estimate (Breitenmoser et al., 2014). This results in different and generally shorter record lengths.

For any remaining candidate pair, the notebook operator is required to make a manual decision based on a summary figure of the duplicates (Fig. 1, step 4). The manual decision encompasses the following choices: keep record #1 and remove record #2 (and vice versa); keep both records; remove both records; or normalize and composite both records. The last choice is useful for individual records which were either composited by the original data producer, or represent a site-local concentrated data collection which represents paleoclimatic information from a particular site over time (Mann et al., 1999; Cobb et al., 2013). The detection algorithm also flags cases in which there may be more than two duplicate records, such that the operator can consistently select the best of the candidate duplicates. In the case of manual decisions, the operator is encouraged to leave a comment regarding the decision made for each candidate pair, in order to ensure that the decision process remains in retrospect comprehensible, consistent and reproducible. This note is later saved in the field duplicateDetail (Table 1).

2.5 Duplicate removal

The final step of the duplicate screening process consists in the removal of the duplicates using notebook dup_removal.ipynb. Here, duplicate records are removed or composited from the database according to the operator's decisions. For the compositing process, the duplicate pair is standardized to z-scores and averaged over their intersecting time period, and their metadata is joined. Note that the duplicate workflow works on identifying potential duplicate candidate pairs, and is thus not optimized for records which appear as duplicates multiple times. However, there are a number of records which appear in more than one database. In particular, there is a significant overlap between p2k, ch2k and iso2k. To make sure no duplicates remain in the final output database, we have therefore implemented a loop of the entire duplicate workflow among the records which appear more than once in the duplicate detection algorithm. The resulting database is finally subject to another duplicate detection workflow to confirm no duplicate records remain.

2.6 DoD2k production

After the removal and/or compositing of duplicates is complete, the duplicate screened database is saved as a pkl file. It is also saved as three comma separated value (csv) files, respectively: year, paleoData_values and metadata (20 fields). A README file contains further information about the duplicate screening process, such as operator name and correspondence details, date and operator's comments. Because the decision process is dependent on the operator's choices, the operator details should always be provided alongside the database to ensure transparency and traceability.

3 Results

3.1 Spatial and temporal coverage

DoD2k v2.0.0 consists of 4781 records (5147 before duplicate screening). It is composed of records from p2k: 969 (1191 records before screening), sisal: 528 (546 records before screening), ch2k: 145 (221 records before screening), iso2k: 410 (435 records before screening), and fe23: 2728 (2754 records before screening). As is true for the parent databases, the spatial distribution (Fig. 2) is heterogeneous and represents mid and high northern latitude terrestrial regions more densely than the oceans, Saharan Africa, subtropical Eurasia, the tropical oceans, the Pacific Ocean and the Southern Ocean.

Figure 2 Spatial distribution of all available proxy records after execution of load, concatenation, duplicate detection and operator supervised decision and removal notebooks. Symbols indicate archive types and legend captions indicate proxy types. “Other” includes n(paleoData_proxy) ≤ 10 for each archiveType. The archiveType “Other” is listed as such in the source originalDatabase.

Although the total number of records available are about 4 times those available in p2k, p2k patterns in temporal availability persist (Emile-Geay et al., 2017, Fig. 3). The total DoD2k record number is highly dominated by the wood (tree) archive, in which the vast majority of observations are tree-ring width (TRW), a subset of which are described in p2k as paleoData_proxy chronologies formed using ARSTAN processing (Cook, 1985), compared to 109 maximum latewood density records (MXD) and 34 δ¹⁸O isotope records. Tree/wood archives in particular represent the most dominant record type after around 1500 CE, when the number of available tree samples rapidly increases. Before this, other archive types are more abundant, especially speleothems, followed by marine and lake sediments and glacier ice. Speleothem records include 16 $\mathrm{Mg}/\mathrm{Ca}$, 136 δ¹³C, 224 δ¹⁸O isotope and 191 growth rate type proxies. Speleothems and marine and lake sediments, in addition to a wider range of observation types included in DoD2k relative to p2k, supply data records on longer timescales, often reaching even further back than the Common Era, and therefore offer a relatively constant coverage over the Common Era, relative to that afforded by p2k (Fig. 3). Although the total number of coral records is also relatively high, these, like records from trees, drop out relatively quickly when moving back in time, and primarily are available in the second half of the most recent millennium. Other rarer archive types include documents and sclerosponges as well as boreholes, hybrid, bivalve, ground ice, mollusk shells, and terrestrial sediments.

Figure 3 Temporal distribution of the DoD2k records by archive type. Records from boreholes, hybrid, bivalve, ground ice, mollusk shells, and terrestrial sediment archives, including all such observations made in those archives, as shown in Fig. 2, are grouped for plotting as “other” for simplicity of presentation. For comparison, right panel shows record availability by archive type from the p2k T-sensitive database (Emile-Geay et al., 2017). Note the log scale for total numbers over time.

3.2 Applications

The development of DoD2k via open-source Python notebooks and functions not only allows for future growth of the aggregate database, but also extraction of subsets of data for specific paleoclimatic analysis. Here we illustrate this with two applications, one utilizing interpretation_variable to filter the data, and the second, utilizing archiveType and paleoData_proxy to provide a target for data modeling (Evans et al., 2013).

3.2.1 Hydroclimate record selection

For the analysis of specific climate conditions, the database can be filtered for interpretation_variable. Approximately two thirds of DoD2k records have non-missing entries. In the Python notebook entitled df_filter.ipynb, we have coded these from the original entries to be listed in DoD2k as “temperature” (T), “moisture” (M), “temperature+moisture” (T+M) and “NOT temperature NOT moisture” ($\sim T\sim M$). For hydroclimatic reconstruction purposes (Kaushal et al., 2022), we can filter the database for M-only or T-only records, but find that only a small subset of records are available and suitable for PCA analysis from a small subset of archives and observation types (see notebooks M_analysis.ipynb, M_analysis.ipynb, results not shown here).

Figure 4 Spatial distribution of the available proxy records with interpretation_variable of moisture (M) or moisture and temperature sensitive (MT). These data are drawn from all 5 component databases forming DoD2k.

For M and T+M records, resulting in 1416 hydroclimate sensitive records, a subset we designate as “MT”, data are available primarily in the Northern Hemisphere, but also in the tropics and at lower latitudes in the Southern hemisphere (Fig. 4). Of these records, 179 are from sisal, 60 are from ch2k, 103 are from iso2k, and 1065 are from fe23. Although the possibility that some p2k records are temperature but also moisture sensitive cannot be excluded, as noted by Emile-Geay et al. (2017), none of these records are taken from the PAGES2k temperature-sensitive database, by its own definition, unless those records were replaced by duplicate records tagged as T+M in the other component databases.

3.2.2 Speleothem δ¹⁸O record modeling

We provide a second application for data modeling within a specific archive and observation type, the oxygen isotopic composition (δ¹⁸O) of cave carbonates (speleothems). We first filter DoD2k for only speleothem δ¹⁸O records. We then simulate δ¹⁸O of speleothem calcite using the sensor model of Partin et al. (2013) as implemented in PRYSM (Dee et al., 2015), which takes as inputs observed δ¹⁸O of precipitation, surface temperature, and mean transit time τ between the surface and the speleothem drip, assuming a well-mixed aquifer source. The archive model is a simple rescaling of the temperature-mediated dripwater δ¹⁸O to account for the difference in scales for the precipitation source (SMOW reference) to the calcite observation (PDB reference). We compare results across the spatial gradients in climatological environmental inputs to assess the null of no difference between simulated and observed speleothem calcite δ¹⁸O (δ¹⁸O_cs and δ¹⁸O_c, respectively). Thus in some ways, this is an extension of the approach taken by Okazaki and Yoshimura (2019) to speleothem calcite δ¹⁸O observations, but without the intermediate step of using an isotope-enabled atmospheric model nudged to observed sea surface temperatures; it is similarly an approach simplified relative to that of Bühler et al. (2021), who compared past millennium (850–1850 CE) climate model simulations of δ¹⁸O_p to air temperature-corrected δ¹⁸O_c observations from 85 calcite speleothem records from 71 sites, using the PRYSM speleothem sensor model to estimate transit times from surface conditions to cave calcification.

Figure 5 Map of locations for which speleothem calcite δ¹⁸O observations are available from the DoD2k within the period 1960–2005 CE.

We perform this exercise in the Jupyter notebook analysis_speleothem.ipynb. When we filter DoD2k for paleoData_proxy and archiveType, we find that the paleoData_values come primarily from the SISALv3 database but also the PAGES2k and Iso2k compilations. Within the comparison period with simulations, 1960–2005 (see below), there are 61 such records available (Fig. 5) from all continents except Antarctica.

For speleothem calcite δ¹⁸O modeling, we import the psm and cfr python packages (Dee et al., 2015; Zhu et al., 2024). We use as inputs the gridded mean annual precipitation amount-weighted climatologically averaged and interpolated terrestrial δ¹⁸O product of Bowen et al. (2005), with estimates taken from the product grid point nearest to the observations (Fig. 5). As this product is based on precipitation amount and δ¹⁸O for 1960–2005, we use this time interval for averaging of other environmental inputs and for the estimation of climatological mean δ¹⁸O of observed speleothem calcite. For surface temperature we use similar nearest gridpoint estimates from the 0.5°×0.5° CRUTS v4.07 surface temperature product (Harris et al., 2020), and average to the same time interval. Because the time interval is short and we average over time, we specify a mean transit time τ of 1 year and we specify constant δ¹⁸O_p for the period of simulation, then average the resulting δ¹⁸O_cs over time. As might be expected the temporal averages of δ¹⁸O_cs are not sensitive to specification of τ.

4 Discussion

4.1 Analysis of the multiarchive, multi-observation MT subset

We perform principal component analysis (PCA) in the Jupyter notebook analysis_moisttemp.ipynb to assess the extent to which there are large scale patterns with and across observational archives and observational types, for the major classes of archives and observations in this subset of data. As opposed to composite averaging, a PCA on standardized proxy observations in their native variable simply identifies patterns of correlation, which could be either positive or negative in sign, between records. Our analysis did so by both archive and observation type in the largest subsets (Table 3). We perform separate PCA on the following five archive and observation subsets of the complete dataset: wood ring width; wood δ¹⁸O; coral δ¹⁸O; speleothem δ¹⁸O and lake sediment δ¹⁸O and δD. Because of the potential for PCA to be influenced by large changes in record availability over time, for each subset we subjectively identify a smaller set of records with a common temporal resolution and a relatively constant data availability over some time period determined by the balance between record length and data availability. For the archive/data subsets described above, the record numbers and temporal availability are given in Table 3. For instance, we gathered TRW records from tree archives at annual time resolution, and found PCA could be performed with 71 records all available over the time interval 1400–1963 for the covariance estimation, and for a time interval spanning 1000–2000, from a 71 record subset. All identified subsets are standardized prior to calculation of their covariance (correlation) matrices and covariance eigendecompositions. Note that information in originalDatabase records for marine sediment archives and δ¹⁸O observations indicate interpretation_variable is either temperature or moisture, not moisture and/or moisture and temperature. Application of the M or MT filter in analysis_moisttemp.ipynb produces only five marine δ¹⁸O records, without a sufficiently long common period of overlap to calculate means and covariances, so we do not perform their PCA.

Table 3 Parameters on the PCA for MT sensitivity subset archive and observation types. Remaining columns are the time resolution at which data were averaged to provide a common resolution across available records, the period of overlap for all records included for covariance estimation, the complete time interval covered by the data subset, and the number of available records entered into each PCA.

Figure 6 Results of the PCA on wood TRW, tree δ¹⁸O, coral δ¹⁸O, speleothem δ¹⁸O and lake sediment δ¹⁸O + δD. (A) Cumulative fraction of explained variance as the sum of normalized eigenvalues corresponding to PCs 1–10. Note: for PCA with dimension >10, only first 10 cumulative fractional eigenvalues are shown. (B, C) 1st and 2nd time series expansion for all 6 PCA. All series with resolution higher than 11 years per timepoint (see Table 3) have been averaged to 11 years per timepoint plotted. PCs of wood δ¹⁸O and coral δ¹⁸O have been multiplied by −1 for the influence of warming on isotopic composition in these archives (Barbour et al., 2004; Konecky et al., 2020a; Walter et al., 2023).

Figure 7 Top: EOF1 loadings across the 5 PCA, corresponding to the first PC shown in Fig. 6. Bottom: as in top panel, except for EOF2 loadings. As in Fig. 6, EOF loadings for tree δ¹⁸O and coral δ¹⁸O have been multiplied by −1.

Results of the PCA are shown in Figs. 6 and 7, respectively. All PCA have a leading pattern which explains from about 40 %–55 % of the variance, with cumulative explained variance from 60 %–75% for PCs 1 and 2 (Fig. 6A). However, perhaps because of differences in observational networks, time resolution and/or covariance estimation interval, there appears to be little agreement between PC1 and PC2 across archive and observation subsets. Although timeseries of the mean of PC1 show, at times, some agreement for certain archives (Fig. 6B), there is no agreement regarding PC2. We acknowledge that there might be differences in sign of linear regression coefficients across, for example, lake sediment δ¹⁸O records (McKay, 2025). In addition, the comparison of sign across the different PCAs shown is problematic, because the sign of each EOF pattern is arbitrary, requiring careful comparison of sign and amplitude of both the spatial patterns and the temporal loadings with the records from which the PCA was developed. We also believe the results may disagree within and possibly across the possible comparison subsets, because there might not be a simple linear mapping or scaling from one environmental variable to the proxy observation, especially for observations which are influenced by both moisture and temperature in different ways and proportions. All this suggests further careful additional analysis is needed before a multi-archive, multi-observational analysis is performed and interpreted.

Figure 7 shows the EOF1 and EOF2 loadings from the 6 independent PCA as a function of their map location. Although there is some degree of regional spatial agreement of EOF sign within and across archives and observations, there are also many instances of disagreement of EOF sign within and across archives and observation types. Because spatial patterns may also be sensitive to observational network and the potential for both T and M influences in this subset, as noted above, we again assess that more analysis within these archives and data types is needed before we can identify large scale patterns across the multi-archive and multi-observation database.

4.2 Speleothem proxy system modeling

We develop and analyze results of the speleothem modeling application in notebook Sanalysis.ipynb. We find an agreement between observed (δ¹⁸O_c,o) and simulated δ¹⁸O_c,s) speleothem calcite isotopic composition that is significantly different from zero (Table 4). This result arises primarily from the regression of calcite δ¹⁸O on precipitation δ¹⁸O, and to a much lesser extent, on the dependence of calcite δ¹⁸O on temperature. The regression of δ¹⁸O_c,o on δ¹⁸O_c,s has a slope of 0.93±0.10 (≪0.0001). The slope is not different from 1:1 by t-test ($t=-\mathrm{0.67}$; p=0.50). The mean difference δ¹⁸O${}_{\mathrm{c},\mathrm{s}}-{\mathit{\delta }}^{\mathrm{18}}$O_c,o is significantly less than zero by t-test (−0.95 ‰) with p=0.04. That value of −0.95 ‰ is about 9 % of the range in observed calcite δ¹⁸O. This mean difference might arise from either the estimated environmental controls or the translation of surface into karst conditions, or the archive model effects, or all of these factors together. Overall, these results demonstrate that simulated and observed mean calcite δ¹⁸O are broadly consistent across a spatial gradient, but with no reduction of true variance across the spatial gradient (Dee et al., 2015; Hu et al., 2017; Okazaki and Yoshimura, 2019; Bühler et al., 2021).

Figure 8 Left: regression of observed calcite oxygen isotopic composition (δ¹⁸O_c) on T, Middle: regression of δ¹⁸O_c on mean annual amount-weighted oxygen isotopic composition (δ¹⁸O_p); Right: regression of simulated (δ¹⁸O_cs) on δ¹⁸O_c. Dashed line is 1:1.

Table 4 Regression diagnostics for regression of observed calcite oxygen isotopic composition (δ¹⁸O_c) on T, on mean annual amount-weighted oxygen isotopic composition (δ¹⁸O_p), and for regression of simulated (δ¹⁸O_cs) on observed calcite oxygen isotopic composition (δ¹⁸O_c). In all cases, N=61 and df = 60.

If including all mineralogies (calcite, aragonite, mixed) from the SISALv3 database, we would obtain a slope of regression of δ¹⁸O_c,s on δ¹⁸O_c,o significantly smaller than unity, and simulated δ¹⁸O_c,s lower than δ¹⁸O_c,o (results not shown). This demonstrates the importance of including that metadata in the original database (Kaushal et al., 2024a) and retaining it in the DoD2k as paleoData_notes. If confirmed with downcore time series and spectral comparisons, this result might be used to either improve the sensor model, by including differential temperature fractionation associated with aragonite vs calcite (Bühler et al., 2021), or construct a more realistic, location-specific archive model (Dee et al., 2015; Bühler et al., 2021).

5 Outlook

Additional challenges that should be carefully considered by adopters and modifiers of the DoD2k begin with the choice and definition of compact dictionary terms by which the aggregation across datasets is performed. We begin with 17 dictionary terms, primarily following a subset of the LiPDverse controlled vocabulary (https://lipdverse.org/vocabulary/, last access: 15 January 2026), but this could and should be expanded, contracted or modified to fit other purposes. For instance, the development of a moisture and temperature subset (Sect. 3.2.1) requires revising original expert interpretation_variable metadata into either moisture or temperature keywords for filtering. There may be cases in which filtering by dictionary terms we have employed could result in ambiguous results. This can be the case when a desired observation type is found in multiple archive types but with different proxy models and climatic interpretations (for instance: $\mathrm{Mg}/\mathrm{Ca}$ in speleothems and foraminifera; δ¹⁸O in wood, marine carbonates, lacustrine carbonates, ice cores). In the case of the SISAL v3 database integration and speleothem analysis (Sect. 3.2.2), there are many other metadata fields than might be useful for successfully leveraging all the information included in that archive (Kaushal et al., 2024a) but which we have neglected in our compact dictionary. More generally, user-supervised choices about selecting and adapting dictionary terms, and managing candidate duplicates may vary according to the application of the DoD2k.

6 Data availability

Databases used to construct the DoD2k are available online (https://lipdverse.org/Pages2kTemperature/2_2_0/, last access: 15 January 2026, updated from Emile-Geay et al., 2017; https://lipdverse.org/CoralHydro2k/1_0_1/, last access: 15 January 2026, updated from Walter et al., 2023; https://lipdverse.org/iso2k/1_1_2/, last access: 15 January 2026, updated from Konecky and McKay, 2020; https://doi.org/10.25921/8HPF-A451, Franke et al., 2022b; https://doi.org/10.5287/ORA-2NANWP4RK, Kaushal et al., 2024b). The DoD2k database is available on GitHub at https://github.com/lluecke/dod2k (last access: 15 January 2026), Zenodo at https://doi.org/10.5281/zenodo.15676255 (Luecke et al., 2026), and at the NOAA/NCEI World Data Service for Paleoclimatology (https://www.ncei.noaa.gov/access/paleo-search/study/41981, last access: 15 January 2026) with https://doi.org/10.25921/sptp-g618 (Evans et al., 2025).

7 Code availability

Jupyter notebooks, Python functions, scripts and tutorials are on GitHub, at https://github.com/lluecke/dod2k (last access: 15 January 2026), with https://doi.org/10.5281/zenodo.15676255 (Luecke et al., 2026).

8 Conclusions

We have developed a compilation of publicly available notebooks and functions, which transform five expert-curated, community sourced Common Era paleoclimate datasets into a database of databases (DoD2k). We demonstrate the utility of the DoD2k with two applications which permit analysis of the DoD2k by paleoclimatic interpretation and by a small subset of archive and observation type. Challenges to this approach include the need to produce a common set of metadata categories, here implemented as dictionary terms, by which to aggregate the component databases within specific groupings by metadata characteristic. The toolkit, consisting of notebooks, utility scripts, and database are available on GitHub. Being user-customizable, they can be used to produce specific DoD2k versions for particular applications.