the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 06 Dec 2024
An expert survey on chamber measurement techniques for methane fluxes
Abstract. Methane is an important greenhouse gas but the magnitude of global emissions in particular from natural sources remain highly uncertain. To estimate methane emissions on large spatial scales, methane flux data sets from field measurements collected and processed by many different researchers must be combined. We hypothesize that considerable uncertainty might be introduced into such data synthesis products by the many different approaches used to collect, process and quality control chamber measurements of methane fluxes within the flux community. Existing guidelines on chamber measurements promote more standardized measurement and processing techniques but to our knowledge, so far, no study has investigated which methods are actually used within the flux community. Therefore, we aimed to identify major differences between the approaches for chamber methane fluxes used by different researchers.
We conducted an expert survey to collect information on chamber-based methane flux measurements, including field sites, research questions, measurement setups and routines as well as data processing and quality control of data. We received 36 responses from researchers in North America, Europe, and Asia which indicated that 80 % of respondents have adopted high-frequency, multi-gas analyzers with most measurement times falling between 2 and 5 minutes. Most but not all of the respondents use recommended chamber designs, including such as airtight sealing, fans, and a pressure vent. We asked about the participants’ approach to quality control and presented a standardized set of methane concentrations from observed flux measurements, then included this information for flux calculations. The responses showed broad disagreement among the experts on processes resulting in nonlinear methane concentration increases. Based on the expert responses, we estimated an uncertainty of 28 % introduced by different researchers deciding differently on discarding vs. accepting a measurement when processing a representative data set of chamber measurement. Different researchers choosing different time periods within the same measurement for flux calculation caused an additional uncertainty of 17 %. Our study highlights the need to understand drivers of the patterns visible from high-resolution analyzers and standardized procedures and guidelines for future chamber methane flux measurements. This is highly important to reliably quantify methane fluxes all over the world.
- Preprint
(4996 KB) - Metadata XML
-
Supplement
(3973 KB) - BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on essd-2024-381', Eyrún Gyða Gunnlaugsdóttir, 08 Jan 2025
It is a very interesting read and a great approach to the important task of unifying methods of chamber measurements for methane, an important addition to fill in a gap of knowledge. The article shows high quality work and data collection. It further on provides a starting point for future researchers to address and reduce the uncertainty of different types of measurement setups or processing of their data.
Easy to read for the most parts. Tended to be a bit text heavy where a table or a figure could have been presented in order to shorten the text. Which in turn would make it easier for the reader to find the results in a more concise manner.
Technical corrections:
Very few cases of 4 in CH4 not being subscripted and usage of both L and l for litre, would be good to unify that.
- Line 55 CH4, need -> needed
- Line 83 CH4
- Line 95 “Our study raises awareness for” Do you mean that the study raises awareness of
reducing the uncertainties etc? I am not following this sentence fully
- Line 129 l -> L
- Line 259 l -> L
- Line 300-04 A bit long sentence that could be shortened or divided into two.
- Line 375 VQC – I do not see an explanation for this abbreviation
- Line 377 mgCH4, space between mg and CH4
- Line 380 not -> nor
- Line 929-32 A bit lengthy sentence
- Line 941 Text not indented for a new paragraphChapter 3.1.3. This chapter would benefit from shortening the text. The schematic figure (4) at next page is very informative and already gives great info.
Figures 4 and 5 are excellent and give a great overview over this part of the results from the survey.
Chapter 3.1.4. This chapter would benefit from shortening the text and adding a descriptive figure/table that can be referred to.
Table 1. The table is taking up quite a lot of space, reaches over to the next page. The participants where all is NR/NA (participants 11, 12, 13 etc) can they be removed from the table and an explanation in the text why they were removed? Then the parts of the table that went to the next page can maybe be squeezed in to this page.
Also in the red boxes at the top of the table, some words are in two lines. Nonlinea – r. Insonsis – tent. The top row is taking up a lot of space and words are cut in the wrong places. The table otherwise is very informative and makes it easy to find the results in a short amount of time.Table 2. Very explanatory and easy to read for concrete results.
Chapters 3.2.2. – 3.2.8.
Figures show the results very well, easy to read. The figures do take up a lot of space. Can they be portrayed a bit smaller? With a less wide x axis, and even 4x3 subplots together in one page? If these figures could all be put into one or two pages, that would be great. Maybe the bigger figures could be in the appendix. Histogram’s y-axis says ”count”, could that rather be participants?
These chapters are quite text heavy, they would all benefit from shortening the text and adding a descriptive figure/table that can be referred to.Table 3. Very difficult to read the table. Could the table be horizontal instead of vertical? Then more text can fit there. The table might also fit better in the appendix. Text could be indented to the left.
I can see that a lot of work and effort has been put into this manuscript, which was a pleasure to read. Good luck with the last steps.
Citation: https://doi.org/10.5194/essd-2024-381-RC1 -
AC1: 'Reply on RC1', Katharina Jentzsch, 16 Jan 2025
Dear Eyrún Gyða Gunnlaugsdóttir,
Thank you very much for your detailed comments which will greatly help us to improve the readability and accessibility of the information given in our manuscript.
In the following, please find our point by point responses (roman text) to your comments (italicized).
It is a very interesting read and a great approach to the important task of unifying methods of chamber measurements for methane, an important addition to fill in a gap of knowledge. The article shows high quality work and data collection. It further on provides a starting point for future researchers to address and reduce the uncertainty of different types of measurement setups or processing of their data.
Easy to read for the most parts. Tended to be a bit text heavy where a table or a figure could have been presented in order to shorten the text. Which in turn would make it easier for the reader to find the results in a more concise manner.
Thank you for this valuable comment. We agree that some parts of the description of the data set are unnecessarily lengthy. We will consider your specific suggestions given below for shortening the text and revise the entire text to make it more precise and easier to follow.
We will address all technical errors that you pointed out in a revised version of the manuscript.
Chapter 3.1.3. This chapter would benefit from shortening the text. The schematic figure (4) at next page is very informative and already gives great info.
We agree that the text is largely repeating the information that is already given in the figure. We will revise and shorten the text to avoid unnecessary repetitions.
Figures 4 and 5 are excellent and give a great overview over this part of the results from the survey.
Chapter 3.1.4. This chapter would benefit from shortening the text and adding a descriptive figure/table that can be referred to.
Thank you for this comment which supports a possible revision of the chapter that we were already considering. We suggest that we summarize the findings of this chapter in a flowchart showing the general process of flux calculation and quality control, complemented by the specific implementations by the survey participants and their frequencies of use. This will allow us to shorten the text and make the key results more easily accessible to the reader.
Table 1. The table is taking up quite a lot of space, reaches over to the next page. The participants where all is NR/NA (participants 11, 12, 13 etc) can they be removed from the table and an explanation in the text why they were removed? Then the parts of the table that went to the next page can maybe be squeezed in to this page.
Thank you for this valuable comment that will greatly improve the readability of this table. You are right – the data from participants who did not give any responses in the visual quality control part of the survey was not used in our uncertainty assessments and therefore does not need to be shown in this table. We will remove the respective rows from the table and add an explanation to the table caption. We are furthermore planning to move the detailed version of table 1 into the supplement and integrate the key results from this table into an extended version of table 2.
Also in the red boxes at the top of the table, some words are in two lines. Nonlinea – r. Insonsis – tent. The top row is taking up a lot of space and words are cut in the wrong places. The table otherwise is very informative and makes it easy to find the results in a short amount of time.
We will correct the hyphenation in the first row and consider shortening the descriptions of the measurement classes in the table to reduce the space that is taken up by the first row.
Table 2. Very explanatory and easy to read for concrete results.
Chapters 3.2.2. – 3.2.8.
Figures show the results very well, easy to read. The figures do take up a lot of space. Can they be portrayed a bit smaller? With a less wide x axis, and even 4x3 subplots together in one page? If these figures could all be put into one or two pages, that would be great. Maybe the bigger figures could be in the appendix. Histogram’s y-axis says ”count”, could that rather be participants?We agree that these figures are taking up a lot of space and that the resolution of the x-axis can be reduced. We will try out ways to reduce the size of the figures and to combine them into one or several multipaneled figures. We will consider moving chapters 3.2.2 to 3.2.8 including the figures into the supplement. We will furthermore change “count” to “number of participants” on the y-axis of the histogram.
These chapters are quite text heavy, they would all benefit from shortening the text and adding a descriptive figure/table that can be referred to.
Thank you for this comment. We will extend Table 2 to include all described measurement classes and move the current sections 3.2.2 to 3.2.8 and the respective figures into the supplement.
Table 3. Very difficult to read the table. Could the table be horizontal instead of vertical? Then more text can fit there. The table might also fit better in the appendix. Text could be indented to the left.
We agree that this table is difficult to read. We will consider changing the orientation of the table, left-align the text, and choose a table design that enhances readability.
Citation: https://doi.org/10.5194/essd-2024-381-AC1
-
AC1: 'Reply on RC1', Katharina Jentzsch, 16 Jan 2025
-
RC2: 'Comment on essd-2024-381', Inge Wiekenkamp, 09 Jan 2025
I have read the manuscript “An expert survey on chamber measurement techniques for methane Fluxes “, written by Katharina Jentzsch and co-authors and considered for publication in ESSD. The article provides a new and unique dataset with expert survey results that evaluate the techniques scientists use to obtain, process and evaluate chamber-based methane fluxes at the field scale. This questionnaire does not only evaluate the specific chamber-based measurement method (system setup etc.), but also evaluates the techniques that are used to process chamber measurements to ultimately obtain the reported methane fluxes. I think that, even though the number of participants is relatively low (as compared to general surveys), the amount of information obtained from the different questions is really nice and contain a lot of information. I think that the expert questions related to different measurement “scenarios” are very interesting and will probably really help the community to understand what discrepancies there are related to data processing and interpretation etc.
Although I really think the manuscript is well written, prepared and interesting, and I also see that the data set is valuable, the current manuscript format is (from my point of view) more similar to the format of a research paper and could be adjusted here and there (reformulated/ restructured, mainly in results, discussion and conclusion section) to really fully fit in ESSD as a data set article. I also was wondering if one important key message is not just that it really could make sense to make raw or quality controlled (with quality id’s) measurement data available in the future and to publish that for chamber-measurement articles? This would make it at least possible that people that want to use a large set of data from a lot of different areas can use their standardized processing and QAQC themselves to make sure that all the data is treated similarly.
I overall enjoyed reading and have provided my (general and detailed) comments and suggestions below.
General comments:
1) Title of the manuscript: I am wondering if the wording “measurement techniques” covers everything mentioned in the manuscript. When I am thinking about the measurement technique, I generally consider mainly the instrumentation, measurement protocol and measurement principles. I, however, think that the survey also includes a lot of questions related to (1) the evaluation data quality control (QAQC) and the (2) data processing approaches. I am wondering if using an alternative wording, such as “data acquisition pipeline”, “measurement framework” or perhaps “data acquisition framework” would help. Alternative, I can imagine that you could also use a second term in the title, such as “measurement techniques and data handling procedure”. I think that an adjustment here could potentially clarify that the survey was not limited to performing the measurement itself.
2) More focus on describing the dataset and explaining potential use case(s) of this dataset (applications/ limitations): I think the paper has put a lot of effort in explaining what we can learn from the dataset and I can definitely understand that the survey creates an important step for awareness on measurement and workflow streamlining to obtain more comparable methane flux data (from chamber measurements). However, as far as I understand, the ESSD journal has a large focus on publishing these unique and important datasets itself – making it different from a general research publication (where one could also show the results of a survey). According to the ESSD website “detailed analyses as authors might report in a research article, remain outside of the scope of this data journal”. Therefore, I would encourage the authors to be more concise with the results of the analysis (for example with the interpretation of the different graphs, especially from Figure 4 onwards) and put more focus on the description of the dataset itself.
3) Recommendation on required methods information in research papers using chamber measurements: As understood, the authors are often talking about making the gap smaller between approaches used in the community. At the same time, they state that details are often missing. Perhaps this publication is also a good place to make a statement on what information about measurement design, QAQC and processing is required in research paper manuscripts (to perhaps also have an uncertainty estimation when combining sets of measurements (based on your survey). I could imagine this to be valuable for the community.
4) Representativeness: I think one important element (that has to be clear in the manuscript and from the data set), where the authors need to elaborate on, is stating whether the people that are interviewed are a qualitatively good “representation” of the community. In other words – how would you judge the quality of the data set, based on the responses you had? What are the pros/ cons and what are the limitations of the 36 responses?
5) Choice for type of question: In your questionnaire, you have made clear choices for using specific types of questions. Surely, the type of question also affects the information you can obtain from the survey (e.g. https://www.soscisurvey.de/help/doku.php/en:create:questiontypes ). It would be great if the authors could elaborate on the choice for the specific question-format they have chosen. The authors have, for example, not used any Likert scale question and have used yes/no questions a lot. Consider also adding some information about these question choices into the manuscript. If the question design has specific pros/ cons, limitations, this could also be discussed in the manuscript.
6) Methods – Use of chamber measurement data: In this questionnaire and data set, measurement from your field campaigns were used for the evaluation of flux processing were used. The authors have explained that they have used a lot of different types of measurement scenarios that can occur. Perhaps the authors can elaborate on (1) the representativeness of the 12 measurement “scenarios” that were chosen (any specific measurement case missing/ not available/ not occurring in your ecosystem/ not present for your measurement system) and (2) the representativeness of this study site for the evaluation of the fluxes and their uncertainty. I can, for example, imagine that the quantitative numbers (and absolute fluxes) would be very different for tropical wetlands.
7) Results, Discussion, Conclusion: I think the results should be more focused on giving an overview over the dataset and what the dataset shows/describes (see also comment 2). Especially the measurement example results (including all figures related to this) make the paper quite lengthy in the end. I suggest shortening here (one could potentially put some of the result figures in the appendix). I would especially consider shortening from Figure 6 and section 3.1.4 onwards.
I am also wondering if, instead of having a very detailed discussion section is appropriate in such dataset paper. Instead of following the typical research paper setup and coming up with a full discussion about the results, I would suggest to stick more to demonstrating what the data set is about, what it can be used for, messages it can convey, etc. I could imagine that a separate paper in the form of an opinion paper or the like could be made to create new guidelines/ setting up certain rules and refer to the current data set.
Also consider all the above-mentioned comments for the conclusion section. I think that this section is also written in the shape of a typical research paper. I would consider rewriting and renaming here to more focus on the data set and what one could use it for.Detailed comments:
- Abstract, Line 11-12, 2nd and 3rd sentence: “To estimate methane emissions … methane fluxes within the flux community.” I could imagine that the transfer to chamber measurements from the 2nd to 3rd sentence is a little abrupt here. While reading I assumed that the authors first spoke about different methane flux measurement methods (e.g., chamber measurement regional airborne mass balance campaigns, tower-based eddy covariance, airborne eddy-covariance, gas column concentration inversion), but the second sentence here is directly directed to chamber measurements. I suggest considering either a direct focus on chamber measurements, or making a clear transfer from all methods to chamber-based fluxes.
- Abstract, line 14 – 15: “Existing guidelines on … within the flux community.” I suggest to be more precise here when defining “flux community”. I think that this community is beyond people that perform chamber measurements and therefore the second part of the sentence might be not very accurate. Also, I would suggest redefining methods (as suggested for the title), as you are not talking about measurement methods to measure methane fluxes, but really about differences in measurement setup, processing, etc. for flux-chamber measurements.
- Abstract, Line 18-19: “We conducted an expert survey … quality control of data.” I would be more precise here as well. Specifically, the segment “to collect information on chamber-based methane flux measurements” could express more precise what you collected (e.g., to collect information on how scientists (?) conduct chamber-based methane flux measurements, etc.…).
- Abstract, Line 21: “… with most measurement times falling between 2 and 5 minutes.” Consider being more specific here with “measurement times” - maybe more clearly mention here that you’re talking about the total measurement time (right?) to obtain (high frequency, > 1 Hz) methane concentrations (and total closure time of the chamber?).
- Abstract, Line 25: “…on processes” Not clear if you are talking about the fact that scientists process the data differently or that they’re thinking about different processes being held responsible (I assume the first). Please consider rephrasing.
- Abstract: I recommend the authors to add a direct link to the data within the abstract. This makes it easy for everyone to access your data directly, even if they only read the abstract.
- Introduction, Line 45 - 48: “achieved …. on the microscale.” Here, the description of gas measurements has a focus on the soil. I suggest to consider plants here as well in the description…
- Introduction, Line 74-75: “establish a more standardized protocol for measurements”. Would be great to mention here if these protocols focus on measurement design and equipment only or if they also discuss the data processing and quality control as well.
- Introduction, Line 90 – 91: “This study aims … data sets.” This comment relates to the general statement (2) that this paper is should focus on presenting the dataset. I would therefore focus here more on the fact that you are presenting this new dataset and what this new dataset is envisioned to provide help with/ answer questions on for the community.
- Introduction, Line 76 – 81: “While guidelines … used by individual researcher”. “At the same time … data set highly uncertain”. Consider rephrasing to really make clear what’s the problem statement and what is currently missing. I think the statement in the last sentence of this block has much more weight (and sounds potentially more serious) as compared to the first. The first sentence sounds as if there are clear guidelines, but they are not followed (not sure if there’s proof for that – might also sound a bit as if people are not well-behaving). The last sentence on the other side sounds rather different – as if there are not yet a complete community-based streamlined set of rules on processing and quality control. Alternative, as mentioned before in the beginning of the review, the interpretation by different scientists could still be different (with motivations specified). However, if the full dataset is available, streamlined flux calculation for larger data sets could still become more feasible/ realistic.
- Methods, Line 103: “Experts were required … of measurements.” I suggest to be more specific here and mention the need for expertise with chamber-measurement for methane fluxes, which I assume was your prerequisite.
- Methods, Line 106: “Altogether, 46 experts were contacted via email”. It would be good to put this number here in perspective. Considering the chamber-measurement flux community, is this a representative group?
- Methods: I think it would also be good to check if, and to mention how your survey and the storage of the provided data set on Pangaea is conform with the EU data protection regulation (just to be on the safe side).
- Methods, Line 112 – 115: “This part of the survey contained 40 questions …image file upload.” I was not sure how to get to the 40 questions (I got to 39) and would recommend to consider rewriting to make it easier to read (and easier to understand).
- Methods, Survey part 2: Last sentence of the paragraph – “We asked them … of the response”. Here it would be maybe good to explain more exact what you asked (start and end seconds). I think this makes it more complete and connects it better to the section where the survey results are shown.
- Section 2.3.2, Line 178 – 180: “For reasons of consistency … were given by the participants”. In these cases, I am wondering how much change a non-linear fit would have given. If the difference in flux would be very large due to the use of a different model, would it not be important to include this? Especially since this could affect (and increase) the uncertainty of the measurement.
- Results, Figure 2b – Maybe it is good to mention in the caption of the figure that participants gave multiple answers regarding their scientific background (causing the total number of responses to be above 36). I could imagine this could also help in other figures.
- Results, Figure 3: I did not find this figure fully intuitive from the start (some of the stand-alone words only made sense after reading the caption of the figure). I also am not 100% sure if this figure is needed, also after having already Figure 2.
- Results, Line 238 – feel like this block of text might need a new heading – e.g. research goals or something the like.
- Results, Figure 4: The total % of the two different sampling techniques does not add up to 100% (26 and 80). Is this again related to the fact that researchers can choose multiple options? In that case, it would be good to mention this in the caption of the figure.
- Table 3 and 1: I think that Table 3 and 1 are not very easy to understand/follow and are both very lengthy in their current form. Considering the fact that this is a data set paper, I suggest that these tables could be simplified with some key messages, adjusted and added to the appendix, or simply be left out (and instead write some important statements as text). The dataset will be there to give the people that are interested detailed information.
Dataset specific comments:
I have looked at the dataset on PANGAEA. Based on the dataset presented on the webpage https://doi.pangaea.de/10.1594/PANGAEA.971695. The dataset is generally in a good shape, but the documentation could be improved. Suggestions that can help to improve the dataset are listed below:
- Name of file(s): I would improve the title of the excel file – A name that directly links to the Pangaea page and/ or the publication would be very nice.
- Personal information: Although the survey results as presented do not contain names etc. and are anonymous, the images of the system setup could potentially relate the science group/ scientist to the answers provided (e.g., thinking of google search that has an image option – specifically if these images are online). Also, the images quickly could provide information about the more specific research area and relate this to PIs/ research groups (people might know each other’s field work areas, equipment etc.). I could imagine that a storage of the different measurement systems separately (e.g., as .png/ .jpeg) could make the results less personally sensitive (thinking about data-privacy), while keeping the information available for others to use.
- Consideration(s) file format: While opening the survey in excel, the response of some questions with Boolean arrays (e.g.: TRUE, FALSE) shifted to the local pc language. Choosing a different data format – such as .csv or .txt could make sure that things, such as change of a date-formats (which happen in excel), do not happen. This also incorporates that you have taken care more of the “I” in FAIR data – it’s probably more interoperable in a different data format. I could image that creating separate files for the different tabs in the excel file would make sense (e.g.: Survey data, VQC data, demographics). When considering a change in file format, it is also nice to make it easy to import into R or Python for example. To simplify the work, the photos that the survey refers to can be stored separately as figures.
- Metadata: In several cases, information about units is not provided (for example in VQC data). It is important to state variable name, description of the variable name, unit and any other important information for such files to be fully used for further research. This includes an explanation on what NA means. Examples of such tables can be found in a paper by Loritz et al., 2024 (https://essd.copernicus.org/articles/16/5625/2024/essd-16-5625-2024.html) for example. This metadata should be added to the Pangaea data repo to make it easy for people to use the data once they download it. I generally also would encourage a short description document that accompanies the dataset (with information about variables, link to article, data license etc.), but this is just a personal recommendation.
- Link to paper and survey: At a later stage, the link to the paper on the Pangaea page would be important to directly switch back and forth between the article, survey questions (supplement) and the dataset resulting from this. The survey questions could also be added to the Pangaea page to make it easier to have everything together.
- Orientation of the questionnaire table: Based on the way I would personally analyze the data (often by one question or a combination of questions), I could imagine that a table orientation where participants are listed as rows and questions are listed as columns could be easier for future processing. I would therefore suggest rotating the table.
Citation: https://doi.org/10.5194/essd-2024-381-RC2 -
AC2: 'Reply on RC2', Katharina Jentzsch, 16 Jan 2025
Dear Dr. Wiekenkamp,
Thank you very much for your detailed comments on our manuscript as well as on the data set itself. Your suggestions will greatly help us to enhance the readability of the article and to clarify our study results. Defining a stronger focus of the article will also make the manuscript better fit for the scope of a “Visions” article in ESSD.
Based on your comments we will make two major adjustments to the manuscript:
- In your comments, you rightly address the deviation of our manuscript format from a classical data set article. The question whether our manuscript fits the scope of ESSD has already been a point of discussion upon submission to the journal. The manuscript has therefore been transferred to “ESS Visions”. We agree that in its current state, our manuscript is somewhat of a mixture between a data set article and a research paper. We therefore suggest that we take full advantage of the opportunity given by an ESS Visions article to focus more strongly on our visions: how to close existing research gaps, i.e. approaches to avoid biases in chamber CH4 flux data sets and to make them more comparable and combinable. We will therefore highlight our recommendation of introducing a chamber flux network/ data platform with a standardized measurements protocol, standardized metadata requirements, and a data quality flagging system. Furthermore, we will propose a further potential use of the data set underlying our survey as a standardized test and training data set for researchers to test their own flux algorithms and compare their results to others.
- To clarify the focus of the manuscript and to enhance the readability of the article, we will move the sections where we discuss the quality control metrics to the supplement, which both reviewers found somewhat lengthy and text-heavy. This includes sections 3.2.2 to 3.2.8, including figures 7 to 18. In the main text we will instead focus more strongly on the uncertainty estimates currently presented in section 3.2.9. To clarify the results, we will consider expanding table 2 to include reasons for keeping or discarding measurements from all measurement classes (linear, nonlinear increase, decrease … in CH4 concentrations) and integrating the key results from Table 1 into this table. We will revise the current detailed version of Table 1 by removing unnecessary rows (as suggested) and move the table to the supplement of the manuscript.
In the following, please find our point by point responses (roman text) to your comments (italicized).
I have read the manuscript “An expert survey on chamber measurement techniques for methane Fluxes “, written by Katharina Jentzsch and co-authors and considered for publication in ESSD. The article provides a new and unique dataset with expert survey results that evaluate the techniques scientists use to obtain, process and evaluate chamber-based methane fluxes at the field scale. This questionnaire does not only evaluate the specific chamber-based measurement method (system setup etc.), but also evaluates the techniques that are used to process chamber measurements to ultimately obtain the reported methane fluxes. I think that, even though the number of participants is relatively low (as compared to general surveys), the amount of information obtained from the different questions is really nice and contain a lot of information. I think that the expert questions related to different measurement “scenarios” are very interesting and will probably really help the community to understand what discrepancies there are related to data processing and interpretation etc.
Although I really think the manuscript is well written, prepared and interesting, and I also see that the data set is valuable, the current manuscript format is (from my point of view) more similar to the format of a research paper and could be adjusted here and there (reformulated/ restructured, mainly in results, discussion and conclusion section) to really fully fit in ESSD as a data set article.Thank you for this comment. As stated above, we are planning to focus more strongly on the conclusions for future research and actions that can be drawn from our study results in order to improve chamber CH4 flux data sets to more clearly frame our results as an ESS Visions article. Furthermore, we will also expand on the other potentially uses (and re-uses) of this dataset.
I also was wondering if one important key message is not just that it really could make sense to make raw or quality controlled (with quality id’s) measurement data available in the future and to publish that for chamber-measurement articles? This would make it at least possible that people that want to use a large set of data from a lot of different areas can use their standardized processing and QAQC themselves to make sure that all the data is treated similarly.
You are right – this is definitely something that we were thinking about but we seem to have ended up not writing it down explicitly. A data control flagging system was also suggested during discussions with other researchers. We will gladly add this point to our conclusions. This addition will help us further in taking full advantage of the opportunities given by an ESS Visions article.
I overall enjoyed reading and have provided my (general and detailed) comments and suggestions below.
General comments:
1) Title of the manuscript: I am wondering if the wording “measurement techniques” covers everything mentioned in the manuscript. When I am thinking about the measurement technique, I generally consider mainly the instrumentation, measurement protocol and measurement principles. I, however, think that the survey also includes a lot of questions related to (1) the evaluation data quality control (QAQC) and the (2) data processing approaches. I am wondering if using an alternative wording, such as “data acquisition pipeline”, “measurement framework” or perhaps “data acquisition framework” would help. Alternative, I can imagine that you could also use a second term in the title, such as “measurement techniques and data handling procedure”. I think that an adjustment here could potentially clarify that the survey was not limited to performing the measurement itself.
Thank you for this very valuable comment. We suggest that we change the title to “An expert survey on chamber measurement techniques and data handling procedures for methane fluxes”.
2) More focus on describing the dataset and explaining potential use case(s) of this dataset (applications/ limitations): I think the paper has put a lot of effort in explaining what we can learn from the dataset and I can definitely understand that the survey creates an important step for awareness on measurement and workflow streamlining to obtain more comparable methane flux data (from chamber measurements). However, as far as I understand, the ESSD journal has a large focus on publishing these unique and important datasets itself – making it different from a general research publication (where one could also show the results of a survey). According to the ESSD website “detailed analyses as authors might report in a research article, remain outside of the scope of this data journal”. Therefore, I would encourage the authors to be more concise with the results of the analysis (for example with the interpretation of the different graphs, especially from Figure 4 onwards) and put more focus on the description of the dataset itself.
As mentioned before, your accurate observation that our manuscript falls between the article types within ESSD has already been a point of discussion with the editors. We are planning to revise the manuscript to further lean into having an ESS Visions article by adding more discussion about the implication and future applications of our study results.
3) Recommendation on required methods information in research papers using chamber measurements: As understood, the authors are often talking about making the gap smaller between approaches used in the community. At the same time, they state that details are often missing. Perhaps this publication is also a good place to make a statement on what information about measurement design, QAQC and processing is required in research paper manuscripts (to perhaps also have an uncertainty estimation when combining sets of measurements (based on your survey). I could imagine this to be valuable for the community.
Thank you for this valuable comment. In the discussion/conclusions, we will explicitly highlight the deviations in measurement setups, data processing, and quality control approaches that are expected to result in significant differences in flux results as information that should always be reported with a flux data set.
4) Representativeness: I think one important element (that has to be clear in the manuscript and from the data set), where the authors need to elaborate on, is stating whether the people that are interviewed are a qualitatively good “representation” of the community. In other words – how would you judge the quality of the data set, based on the responses you had? What are the pros/ cons and what are the limitations of the 36 responses?
We think that although the number of survey participants is relatively small, they are still representative of many of the approaches currently used for chamber CH4 flux measurements. We estimate that the total number of researchers measuring CH4 fluxes with chambers all over the world to be relatively small, maybe including several hundred people. However, you are right in that we should add a section explicitly addressing the representativeness of the survey responses to the discussion section of the manuscript. While the relatively small number of survey participants makes statistical conclusions on how commonly a certain method is used within the flux community highly uncertain, we made sure that the survey participants are rather independent from one another in order to present a large range of measurement and data handling approaches. What the survey does clearly point out is where the main challenges lie: not so much in the chamber setup and instrumentation stage, because agreement is high there (Figure 4), but in this data processing and analysis stage.
From the variety of survey responses it becomes clear that evaluating the representativeness of the respondents remains challenging. One reason for this might be that the chamber flux measurement community remains less organized than the EC flux measurement community and is more fluid because the barriers for entry (cost of analysis) is lower. However, we will add that there are perhaps parts of the community that we did not reach with our survey, for example those working in agricultural ecosystems. We will discuss the potential effect on the survey results of including this part of the chamber flux community, that likely often observes low or close-to-zero CH4 fluxes.
5) Choice for type of question: In your questionnaire, you have made clear choices for using specific types of questions. Surely, the type of question also affects the information you can obtain from the survey (e.g. https://www.soscisurvey.de/help/doku.php/en:create:questiontypes ). It would be great if the authors could elaborate on the choice for the specific question-format they have chosen. The authors have, for example, not used any Likert scale question and have used yes/no questions a lot. Consider also adding some information about these question choices into the manuscript. If the question design has specific pros/ cons, limitations, this could also be discussed in the manuscript.
We highly appreciate your comments on the appropriate handling of survey data and the required information. Thank you for providing the link to this very useful overview of question types. We chose the question types mainly with the goal to make the responses comparable among the participants while still obtaining detailed information on the reasoning for the use of specific measurement and data handling techniques. As pointed out earlier, the limited number of survey participants required a limited number of possible responses to give meaningful results and yes/no questions allowed us to say something about the prevalence of many of the best practices. We will add some sentences to the discussion section of our manuscript. One challenge was that we had somewhat limited examples to choose from as this approach of an expert survey rather than an expert assessment is not so commonly employed (“science of doing science”). We showed here that this approach to surveying experts on their methods can be useful and can recommend that this approach can be a beneficial line of research.
6) Methods – Use of chamber measurement data: In this questionnaire and data set, measurement from your field campaigns were used for the evaluation of flux processing were used. The authors have explained that they have used a lot of different types of measurement scenarios that can occur. Perhaps the authors can elaborate on (1) the representativeness of the 12 measurement “scenarios” that were chosen (any specific measurement case missing/ not available/ not occurring in your ecosystem/ not present for your measurement system) and (2) the representativeness of this study site for the evaluation of the fluxes and their uncertainty. I can, for example, imagine that the quantitative numbers (and absolute fluxes) would be very different for tropical wetlands.
We agree that a paragraph should be added on the representativeness of the Siikaneva data set as our uncertainty assessments strongly depend on the occurrence of different measurement types in the underlying reference data set. An example for a missing measurement class is strong net CH4 uptake which might be observed in drier ecosystems but which did not occur at our peatland site (weak net uptake to zero fluxes were observed and included in this survey). The overall magnitude of the fluxes furthermore affects the impact of deviations in the processing methods used.
7) Results, Discussion, Conclusion: I think the results should be more focused on giving an overview over the dataset and what the dataset shows/describes (see also comment 2). Especially the measurement example results (including all figures related to this) make the paper quite lengthy in the end. I suggest shortening here (one could potentially put some of the result figures in the appendix). I would especially consider shortening from Figure 6 and section 3.1.4 onwards.
We agree that the description/ interpretation of the individual measurement examples in the visual quality control exercise is quite lengthy and this was also pointed out by the other referee. We will therefore extend Table 2 to summarize the most important points mentioned by the survey participants for each measurement class and the sections 3.2.2 to 3.2.8 including figures 7 to 18 to the supplement.
I am also wondering if, instead of having a very detailed discussion section is appropriate in such dataset paper. Instead of following the typical research paper setup and coming up with a full discussion about the results, I would suggest to stick more to demonstrating what the data set is about, what it can be used for, messages it can convey, etc. I could imagine that a separate paper in the form of an opinion paper or the like could be made to create new guidelines/ setting up certain rules and refer to the current data set.
As explained above, instead of turning the manuscript into a clear data paper, we are planning to lean more strongly into the opportunities of having an ESS Visions article by expanding on future implications of our study results, as suggested here with the possible opinion paper.
Also consider all the above-mentioned comments for the conclusion section. I think that this section is also written in the shape of a typical research paper. I would consider rewriting and renaming here to more focus on the data set and what one could use it for.
We will revise the structure of the manuscript to address this point as explained in major reviews comment #1 at the top.
Detailed comments:
Abstract, Line 11-12, 2nd and 3rd sentence: “To estimate methane emissions … methane fluxes within the flux community.” I could imagine that the transfer to chamber measurements from the 2nd to 3rd sentence is a little abrupt here. While reading I assumed that the authors first spoke about different methane flux measurement methods (e.g., chamber measurement regional airborne mass balance campaigns, tower-based eddy covariance, airborne eddy-covariance, gas column concentration inversion), but the second sentence here is directly directed to chamber measurements. I suggest considering either a direct focus on chamber measurements, or making a clear transfer from all methods to chamber-based fluxes.
Thank you for this comment. We will clarify by adding a sentence in between, introducing chamber measurements as one type of flux measurement techniques.
Abstract, line 14 – 15: “Existing guidelines on … within the flux community.” I suggest to be more precise here when defining “flux community”. I think that this community is beyond people that perform chamber measurements and therefore the second part of the sentence might be not very accurate. Also, I would suggest redefining methods (as suggested for the title), as you are not talking about measurement methods to measure methane fluxes, but really about differences in measurement setup, processing, etc. for flux-chamber measurements.
We will clarify that we are only considering chamber-based measurements and the researchers using flux chambers in our study.
Abstract, Line 18-19: “We conducted an expert survey … quality control of data.” I would be more precise here as well. Specifically, the segment “to collect information on chamber-based methane flux measurements” could express more precise what you collected (e.g., to collect information on how scientists (?) conduct chamber-based methane flux measurements, etc.…).
We will rewrite this sentence to clarify that we are investigating how different scientists conduct chamber measurements and handle the resulting data.
Abstract, Line 21: “… with most measurement times falling between 2 and 5 minutes.” Consider being more specific here with “measurement times” - maybe more clearly mention here that you’re talking about the total measurement time (right?) to obtain (high frequency, > 1 Hz) methane concentrations (and total closure time of the chamber?).
We will rephrase the sentence and use the suggested term “closure time of the chamber” instead of “measurement time”.
Abstract, Line 25: “…on processes” Not clear if you are talking about the fact that scientists process the data differently or that they’re thinking about different processes being held responsible (I assume the first). Please consider rephrasing.
We actually meant the latter. We will rephrase the sentence to clarify to something like “… the processes considered responsible for nonlinear concentration increase…”
Abstract: I recommend the authors to add a direct link to the data within the abstract. This makes it easy for everyone to access your data directly, even if they only read the abstract.
Thank you for this suggestion – we will add the link.
Introduction, Line 45 - 48: “achieved …. on the microscale.” Here, the description of gas measurements has a focus on the soil. I suggest to consider plants here as well in the description…
By “soil” we actually just meant that the chamber is placed on the surface. We will clarify that any vegetation is also covered by the chamber and considered in assessment of gas exchange.
Introduction, Line 74-75: “establish a more standardized protocol for measurements”. Would be great to mention here if these protocols focus on measurement design and equipment only or if they also discuss the data processing and quality control as well.
Thank you for this suggestion. We will briefly elaborate on the focus of the individual studies.
Introduction, Line 90 – 91: “This study aims … data sets.” This comment relates to the general statement (2) that this paper is should focus on presenting the dataset. I would therefore focus here more on the fact that you are presenting this new dataset and what this new dataset is envisioned to provide help with/ answer questions on for the community.
We will rephrase the study aim to highlight the nature of the manuscript as an ESS Visions article.
Introduction, Line 76 – 81: “While guidelines … used by individual researcher”. “At the same time … data set highly uncertain”. Consider rephrasing to really make clear what’s the problem statement and what is currently missing. I think the statement in the last sentence of this block has much more weight (and sounds potentially more serious) as compared to the first. The first sentence sounds as if there are clear guidelines, but they are not followed (not sure if there’s proof for that – might also sound a bit as if people are not well-behaving). The last sentence on the other side sounds rather different – as if there are not yet a complete community-based streamlined set of rules on processing and quality control. Alternative, as mentioned before in the beginning of the review, the interpretation by different scientists could still be different (with motivations specified). However, if the full dataset is available, streamlined flux calculation for larger data sets could still become more feasible/ realistic.
This is a very valuable comment. We will revise the section to stress the actual problem that methods are often not documented sufficiently instead of emphasizing what we guess is the case or learned from personal exchange.
Methods, Line 103: “Experts were required … of measurements.” I suggest to be more specific here and mention the need for expertise with chamber-measurement for methane fluxes, which I assume was your prerequisite.
Yes, the word “expertise” is missing here. We will rephrase the sentence accordingly.
Methods, Line 106: “Altogether, 46 experts were contacted via email”. It would be good to put this number here in perspective. Considering the chamber-measurement flux community, is this a representative group?
We will elaborate on our choice of flux experts that we contacted and put their number into perspective considering the entire chamber-measurement flux community of maybe several hundred researchers. We will furthermore add a paragraph later in the manuscript discussing the representativeness of the chosen group of experts.
Methods: I think it would also be good to check if, and to mention how your survey and the storage of the provided data set on Pangaea is conform with the EU data protection regulation (just to be on the safe side).
Thank you for this very important suggestion. The survey itself was officially checked in terms of data protection regulations before application but we will inquire once more about further regulations on the publication of the data.
Methods, Line 112 – 115: “This part of the survey contained 40 questions …image file upload.” I was not sure how to get to the 40 questions (I got to 39) and would recommend to consider rewriting to make it easier to read (and easier to understand).
We will check again the numbers of the questions, rephrase, and also elaborate on the reasoning behind our choice of question types as suggested above.
Methods, Survey part 2: Last sentence of the paragraph – “We asked them … of the response”. Here it would be maybe good to explain more exact what you asked (start and end seconds). I think this makes it more complete and connects it better to the section where the survey results are shown.
We will make this sentence more explicit by mentioning that we asked for the seconds after chamber closure at which the participants would start and end the time interval of the chamber measurement that they would consider for flux calculation.
Section 2.3.2, Line 178 – 180: “For reasons of consistency … were given by the participants”. In these cases, I am wondering how much change a non-linear fit would have given. If the difference in flux would be very large due to the use of a different model, would it not be important to include this? Especially since this could affect (and increase) the uncertainty of the measurement.
This is correct – including nonlinear fits could strongly affect the estimated uncertainty. We have so far decided to use linear fits only, because often the type of nonlinear fit is not specified and because we could not be sure that all participants who would rather use a linear fit actually considered that this was an option in this visual quality control exercise. We will consider using nonlinear fits when suggested in a revised version of the manuscript.
Results, Figure 2b – Maybe it is good to mention in the caption of the figure that participants gave multiple answers regarding their scientific background (causing the total number of responses to be above 36). I could imagine this could also help in other figures.
Yes, we will clarify this in the respective figure captions.
Results, Figure 3: I did not find this figure fully intuitive from the start (some of the stand-alone words only made sense after reading the caption of the figure). I also am not 100% sure if this figure is needed, also after having already Figure 2.
We will reconsider the significance of this figure when revising the manuscript. We considered the ecosystem types where chamber measurements are applied as interesting because they might affect the suitability of specific measurement setups as well as the occurrence of certain measurement classes (development of CH4 concentrations over the time of the chamber closure) and thus the relevance and applicability of certain processing procedures. If keeping the figure, we will add short headings to the three parts of the figure to make its content more readily accessible to the reader.
Results, Line 238 – feel like this block of text might need a new heading – e.g. research goals or something the like.
Yes, we will make this paragraph a separate section on the research goals of the participants that has its own heading.
Results, Figure 4: The total % of the two different sampling techniques does not add up to 100% (26 and 80). Is this again related to the fact that researchers can choose multiple options? In that case, it would be good to mention this in the caption of the figure.
Yes, several participants mentioned that they use both manual sampling and in-line gas analyzers. We will clarify that the researchers could choose multiple options.
Table 3 and 1: I think that Table 3 and 1 are not very easy to understand/follow and are both very lengthy in their current form. Considering the fact that this is a data set paper, I suggest that these tables could be simplified with some key messages, adjusted and added to the appendix, or simply be left out (and instead write some important statements as text). The dataset will be there to give the people that are interested detailed information.
You are right – the significance of these tables depends very much on the final focus of this article as they go beyond a description of the data set. We consider Table 1 as important to support our estimates of the uncertainty introduced through the decision of accepting or rejecting a measurement and through the selection of a time period for flux processing, as the fluxes calculated from the visual QC exercise are not included in the survey responses themselves. We will remove the rows of participants who did not participate in the visual QC exercise and consider moving the table to the supplement. We will furthermore change the orientation of Table 3 and left-align the text as suggested by the other referee to enhance its readability.
Dataset specific comments:
I have looked at the dataset on PANGAEA. Based on the dataset presented on the webpage https://doi.pangaea.de/10.1594/PANGAEA.971695. The dataset is generally in a good shape, but the documentation could be improved. Suggestions that can help to improve the dataset are listed below:
Name of file(s): I would improve the title of the excel file – A name that directly links to the Pangaea page and/ or the publication would be very nice.
We will revise the file name to include authors, title, and year of publication.
Personal information: Although the survey results as presented do not contain names etc. and are anonymous, the images of the system setup could potentially relate the science group/ scientist to the answers provided (e.g., thinking of google search that has an image option – specifically if these images are online). Also, the images quickly could provide information about the more specific research area and relate this to PIs/ research groups (people might know each other’s field work areas, equipment etc.). I could imagine that a storage of the different measurement systems separately (e.g., as .png/ .jpeg) could make the results less personally sensitive (thinking about data-privacy), while keeping the information available for others to use.
Thank you for pointing this out. While we have asked every survey participant for permission to upload their photos, we prefer your idea to store the photos separately from the other information.
Consideration(s) file format: While opening the survey in excel, the response of some questions with Boolean arrays (e.g.: TRUE, FALSE) shifted to the local pc language. Choosing a different data format – such as .csv or .txt could make sure that things, such as change of a date-formats (which happen in excel), do not happen. This also incorporates that you have taken care more of the “I” in FAIR data – it’s probably more interoperable in a different data format. I could image that creating separate files for the different tabs in the excel file would make sense (e.g.: Survey data, VQC data, demographics). When considering a change in file format, it is also nice to make it easy to import into R or Python for example. To simplify the work, the photos that the survey refers to can be stored separately as figures.
Thank you for this advice. We will change the file format to make it more interoperable and publish the image files separately.
Metadata: In several cases, information about units is not provided (for example in VQC data). It is important to state variable name, description of the variable name, unit and any other important information for such files to be fully used for further research. This includes an explanation on what NA means. Examples of such tables can be found in a paper by Loritz et al., 2024 (https://essd.copernicus.org/articles/16/5625/2024/essd-16-5625-2024.html) for example. This metadata should be added to the Pangaea data repo to make it easy for people to use the data once they download it. I generally also would encourage a short description document that accompanies the dataset (with information about variables, link to article, data license etc.), but this is just a personal recommendation.
We will add both metadata and description files to the Pangaea publication to avoid misinterpretations and clarify the relation of the uploaded files.
Link to paper and survey: At a later stage, the link to the paper on the Pangaea page would be important to directly switch back and forth between the article, survey questions (supplement) and the dataset resulting from this. The survey questions could also be added to the Pangaea page to make it easier to have everything together.
A link to the final paper will be added to the Pangaea webpage upon publication. While a short version of most questions is given in the first column of the excel spreadsheet, we will add the complete overview of the survey questions that is currently included as a supplement to the article preprint to the Pangaea page in .pdf format.
Orientation of the questionnaire table: Based on the way I would personally analyze the data (often by one question or a combination of questions), I could imagine that a table orientation where participants are listed as rows and questions are listed as columns could be easier for future processing. I would therefore suggest rotating the table.
We will reconsider the orientation of the table, although we have intentionally choses this orientation to allow for grouping the survey questions into the categories of “Demographic information”, “Flux measurement sites”, etc. that are also used in the manuscript.
Citation: https://doi.org/10.5194/essd-2024-381-AC2
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 220 | 58 | 9 | 287 | 46 | 6 | 6 |
- HTML: 220
- PDF: 58
- XML: 9
- Total: 287
- Supplement: 46
- BibTeX: 6
- EndNote: 6
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1