It is a timely and valuable dataset that can be useful for not only building classification but training a multi-modal AI model to a better understanding of the function of urban buildings as well as the underlining motivation of human activities and movements. To the best of my knowledge, it is the first multimodal dataset dedicated to building function classification that offers 26 distinct, fine-grained categories. This is a significant improvement over existing schemes that often mirror coarse land-use classifications. The study challenges the conventional belief that large models cannot handle multimodal spatial data, which is a high-quality contribution. Overall, the paper is well-structured, the methodology is sound, and the dataset indeed fills a clear gap in the Earth System Science community. Some issues in below should be addressed before publication.

1. The dataset is exclusively sampled from New York City rather than from a broader geography. The authors acknowledge that conclusions may be subject to urban bias, as variations in model performance and error distributions might occur in cities with different layouts. A more in-depth discussion of impacts of the geography bias on the inference of models as well as the results of the baselines should be provided.
2. One important contribution of the dataset is the rich function Table 3 lists existing building datasets without indicating the number of building function categories. I would suggest adding # of categories in the table.
3. The method section describes a workflow. I suggest adding a workflow chart at the beginning of the method section to make the readers easily understanding the process.
4. It is good to have an evaluation for the performance of the large models handling the multimodal dataset. However, you should give a logical and clear justification of the reasons of doing the evaluation. Why did you only involve those large models? Any specific reasons of not testing popular machine learning and deep learning methods?
5. The authors claim that the sampling processing of the building footprints significantly reduces geospatial bias. In the BuildingSense section, a map showing the geographical distribution of the samples should be provided. It can help readers better understand the data quality in terms of geographical coverage.

This paper constructs a multimodal fine-grained dataset for building function classification, covering remote sensing imagery, street-view images, POI data, and building attributes for over 34,000 buildings in New York City. Through a systematic literature review, it identifies current methodological bottlenecks in building function classification and the scarcity of datasets, while revealing both the potential and research gaps of large models in this task. The subsequent benchmark experiments with large models demonstrate the promise of multimodal fusion for understanding building functions, challenging prevailing views in the field. The dataset construction process is solid and includes appropriate quality-control measures. The paper is timely and valuable, and falls within the scope of ESSD.

However, there are some issues that need to be addressed before publication:

1. As the first contact with readers, the abstract should briefly highlight the dataset's scale (e.g., number of buildings, modalities included).
2. I noticed that the street-view images in the published dataset do not have the target buildings annotated. This will undoubtedly lead to “Spatial relationship errors” in the current study and similar issues in future models trained on this dataset. Could the authors explain the reasoning behind this?
3. There is a typo in Equation (1): "arctan" should replace "acrtan"
4. In the POI section, adding a field table would improve readability.
5. There are spelling inconsistencies for "BuildingSense" throughout the manuscript (e.g., line 165).
6. While large models have surpassed traditional methods in reasoning on complex tasks and can significantly enhance the interpretability of results, it is still necessary to explain why traditional deep learning models were not evaluated on this building function dataset.

7. Please add crucial information about “residential category statistics scaled by a factor of 10” in the title of Figure 7 to prevent misinterpretation.