This study proposed a crop yield prediction benchmark for sub-national crop yield forecasting.  The authors did a good job to build the dataset and the data sharing platform. However, there are some fundamental issues with the study design and protocols that need to be addressed before it can be considered for publication:

• Novelty: The novelty of this study is not entirely clear. Based on the reviewer's previous experience reviewing/reading benchmark dataset papers, researchers typically contribute either newly collected datasets—often the result of years of fieldwork or labor-intensive data labeling—or propose novel modeling approaches that produce grid-level maps with global coverage. Importantly, such studies also provide new insights derived from their unique datasets. In contrast, the current study relies on publicly available yield statistics, processes commonly used data sources, and generates predictors following standard protocols. Both the yield data and predictors have already been widely used in previous research. While compiling them in one place is useful, it is difficult to identify what is truly novel about this work.
• Static crop masks: Crop masks play a crucial role in processing yield predictors, as they help eliminate irrelevant pixels and reduce noise in the data. However, this study applies a static crop mask across multiple years, which is problematic. This approach does not align with the standards typically expected in benchmarking studies.
• Quality Control and Uncertainty analysis: Government-reported statistics are not always accurate, and the quality of data can vary significantly across countries. Therefore, it is essential to perform quality control and/or assess the uncertainty associated with the data samples—an important step that appears to be missing in the current study.

More specific comments are given below:

1. “Crop yield forecasts are produced by both private entities and government institutes using field surveys, process-based crop models, statistical regression and machine learning (Basso and Liu, 2019; Schauberger et al., 2020; Paudel et al., 2021; Gavasso- Rita et al., 2023)”: It would be helpful if the authors clearly indicated which citations are associated with each specific method. Also, it is not clear what is the difference between statistical regression and machine learning.
2. Line 35: It is biased to only give the example of EU while ignoring other major crop production regions (China, US, South America)
3. Line 59: “Benchmark datasets must cover a wide variety of regions and countries”: The rationale behind this requirement is unclear. Why is it essential for benchmark datasets to span such a broad geographic scope? I think benchmark datasets are task specific. From the classic computer vision dataset ImageNet to the remote sensing datasets (the UC-Merced dataset (Yang and Newsam, 2010), the WHU-RS19 dataset (Xia et al., 2010), the AID dataset (Xia et al., 2017)), they focus on specific classes/regions/tasks. For the same reason, I think a dataset focusing on U.S. only or China only can still be regarded as benchmarks.
4. Line 110: “we engaged a diverse community of researchers to weigh the benefits and limitations of data sources for each type of data necessary to produce crop yield forecasts” Could the authors be more specific on the decision and quality control processes? What specific benefits and limitations have been considered?
5. Line 115: “The most relevant weather variables for crop yield forecasting are temperature, solar radiation, and precipitation” It is controversial. Some studies have claimed that VPD and ET are more informative than temperature and precipitation in terms of yield prediction.
6. Line 118: Why was AgERA5 selected? I think there are a lot of alternative choices (PRISM, Gridmet, TerraClimate …) that have been used in yield prediction studies.
7. Line 127: I think SSM and RSM are highly correlated.
8. Line 145: It is not clear why only fPAR and NDVI are chosen. NDVI has been known for its issue of saturation and there are quite a few alternatives (EVI, GCVI).
9. Line 149: Why using the eight-day composite?
10. “Predictor data and yield statistics often differ in spatial and temporal resolution, requiring further processing to align them effectively” How would the mismatch between data sources increase the uncertainty and impact the quality of the predictors?
11. Line 237 “CY-Bench currently includes predictor data up to and including 2023.” What is the starting year?
12. Figure 3-4: Since the CY benchmark dataset focuses on the sub national yield statistics, it is misleading to color the whole counties, instead of color the sub-national units with the yield records. For example, there is no corn/wheat in US Alaska.
13. “The crop masks and crop calendars included in CY-Bench are static, i.e. they do not reflect yearly changes.” It is a very critical issue. The cropland experiences dramatic changes from year to year, even at the subnational level. By using a static crop mask, the predictors can be totally wrong in early years.
14. On the shared Github Leaderboard, there are quite a few counties in which the ML models achieved poor accuracy (negative R2). Given that, how would the authors justify the input features have been well processed, or the uncertainty in the yield data has been well controlled? Could the authors explain if it is safe to use a benchmark datasets whose benchmark accuracies are that low? 

This study proposes CY-Bench, a global, sub-national benchmark for in-season, pre-harvest crop-yield forecasting of maize (covering 38 countries) and wheat (covering 29 countries). The data are collected and curated from large-scale open-access sources. The data are potentially useful for developing and evaluating machine learning models for crop yield forecasting and other Earth system related tasks. In general, CY-Bench is novel and useful for the machine learning and agricultural communities. The following comments should be considered before formally publishing it:

1. As CY-Bench is proposed for developing and evaluating data driven models, you should discuss and compare the performance of the models you have benchmarked in the main text, despite a table of model performance is provided in the code repo. You should also discuss whether there are different findings (of benchmarking these models) compared to existing datasets.

2. Your benchmarking results (as shown in the code repo) displays large variations across different regions. For example, Maize (CN) achieves low NRMSE (8.78) and close-to-one R2 (0.81), while Maize (DK) reaches much higher NRMSE and all very negative R2. This problem can be caused by the models you train or the dataset itself (e.g., quality of the data, predictors you choose), and should be sufficiently discussed in the paper.

3. Following the previous comment, I recommend adding a note for each region (or a group of regions) to guide potential users on the specific precautions needed when working with its data (e.g. quality concerns, noise or any other risks).

4. The size of data in each region should be indicated.

5. Lines 275 - 280, you should discuss why LOYO is better than random sampling, especially if random sampling was previously more commonly used as you mentioned. As you have noted in limitations, LOYO is a compromise for small datasets; is there a better way for regions with more data?

6. In Section 2, how the data are harmonized should be described with more details, instead of simply mentioning it (e.g., Lines 180 - 185 "as they follow a harmonization procedure developed by ..." and Lines 185 - 190 "The data was compiled and harmonized to account for ...").

7. Following the previous comment, you may consider describing more details on how you collect and curate data from a large number of different sources, as this is a major methodological contribution of your study and will be useful for other researchers.