Our objective was to build a dataset that closely represents the true distribution of surface water features using only 100 samples. A representative dataset enables testing the spatial generalizability and accuracy of surface water extent products. However, achieving a true representation is nearly impossible . We approached this challenge by sampling from different biomes, as defined by , which encompass various climates and land conditions, giving a better chance of providing high variance within samples.

We employed a stratified random sampling strategy to ensure the representativeness of our dataset. First, we created a 2 km buffer around global river and lake shapefiles provided by the using QGIS (Quantum GIS). We then clipped these buffers with the shapefiles of each of the 14 biomes. Within each biome, we randomly placed 50 points using QGIS's random point generator and selected at least 5 of them as samples.

To address the various contexts in which surface water exists, we randomly selected additional samples from urbanized regions , braided rivers, and coastal regions. Urban areas are spatially heterogeneous, often resulting in increased complexity for water detection. We also separately sampled from lakes and rivers to ensure a balanced representation of both water body types. Braided rivers and coastal areas were included.

Figure  shows the number of samples for each biome, while Fig.  illustrates the global spatial distribution of the samples. The numbers of samples from “Tropical and Subtropical Dry Broadleaf Forests” and “Tropical and Subtropical Coniferous Forests” were limited due to their smaller areal coverage. Approximately two-thirds of our labels are from rivers and the remaining one-third are from lakes. We sampled a larger portion from “Deserts and Xeric Shrublands” (16 samples) because water extraction methods generally perform worse in these regions, especially when using radar imagery .

The temporal distribution of our samples spans from 2021 to 2023, covering different seasons to capture seasonal variations in surface water extent. While our sampling strategy aimed to maximize representativeness within the constraints of labeling resources, we acknowledge that the limited number of samples (100) may not fully capture all global surface water variations.

During the sampling process, we implemented quality control measures to ensure that the selected locations were suitable for labeling and analysis. We downloaded the PlanetScope scene for each location, divided the scene into 1024×1024 sized images, and then selected the image that contained sufficient water and no cloud cover.

After selecting 100 locations based on our sampling strategy, we downloaded 8-band, 3 m resolution SuperDove PlanetScope (PS) imagery from 2021 to 2023 using our access to the NASA Commercial SmallSat Data Acquisition (CSDA) program. As our objective was to evaluate most medium-resolution satellite sensors, including Sentinel-1 (S1), we ensured that the failure of the Sentinel-1B satellite, on 23 December 2021, did not create a large temporal gap between the label and the last available scene from the satellite. For locations only covered by Sentinel-1B and not Sentinel-1A, we acquired PS scenes before the Sentinel-1B failure date.

During the scene selection process, we excluded areas with perennially frozen water. If a location contained seasonal ice, we replaced that PS image with a summer image when the water was not frozen. This approach ensured that our dataset focused on liquid water surfaces, which are more relevant for surface water extent mapping.

From each larger PS scene, we extracted a 1024×1024 pixel image, covering an area of approximately 9.4 km2. We chose 1024×1024 pixel images to ensure sufficient pixels and spatial context for comparison with medium-resolution imagery (e.g., Landsat, Sentinel). For instance, a 30 m Landsat image corresponding to our labels would have around 100×100 pixels, while a 10 m Sentinel image would have approximately 376×376 pixels. Figure  showcases two examples of the PS images selected for labeling, displayed in false-color composite (near-infrared, red, and green bands).

We used high-resolution 3 m PlanetScope (PS) data for labeling, ideal for the evaluation of lower-resolution satellite products such as Sentinel-1 (S1), Sentinel-2 (S2) at 10 m, or Landsat sensors at 30 m.

The labeling was performed by experienced analysts to distinguish between three classes: water, low-confidence water, and non-water. The water class represents areas with a clear presence of water, while the low-confidence water class marks pixels where the presence of water is uncertain but probable. The non-water class encompasses all other land cover types. To assist the labelers, we provided true-color composite (TCC) and false-color composite (FCC) images using the near-infrared, red, and green bands for each sample.

In cases where the presence of water was unclear in the PS imagery, we cross-referenced them with higher-resolution basemaps from Bing and Google. Unresolved features were assigned to the low-confidence water category, ensuring that the water class only includes pixels with a high degree of certainty. During the evaluation process, the low-confidence water class can be excluded or added to the water category as necessary.

To streamline the labeling process and ensure the creation of high-quality labels, we utilized the Labelbox platform , which provides efficient tools for data annotation. After the initial labeling, we performed several rounds of quality checks on each label to maintain accuracy and consistency across the dataset.

In total, we labeled 100 images, each with a size of 1024×1024 pixels, covering a total surface area of 940 km2. The labeling process, including quality control, took approximately 2 h per image, resulting in a total of 200 h of work. The labeled surface water accounts for nearly 250 km2 of the total area. Each label is assigned a unique sample ID (SID) ranging from 1 to 100 and includes the date (YYYYMMDD) of the PS image used for labeling.

We labeled a total of one-hundred 1024×1024 PS images at 3 m, with the overall class distribution covering 24.9 % of the total surface area, low-confidence water covering 1.2 %, and the rest (73.9 %) being non-water (Fig. ). The distribution of water pixel percentages for each individual label, as displayed in Fig. , demonstrates that most labels contain less than 50 % water pixels by design, with the mean water surface area per label being 26.10 km2. This focus on having more non-water area enables better delineation of water boundaries, as the water class itself tends to be more homogeneous and therefore less complex from both labeling and mapping perspectives.

As mentioned previously, our labeled dataset covers water surface areas across different biomes (Table ). The mean percentage of water content per label varies substantially between biomes, from a low of 5.29 % for “Mediterranean Forests, Woodlands and Scrub” to a high of 42.95 % for “Temperate Grasslands, Savannas and Shrublands”. This demonstrates the diversity of landscapes and water coverage captured in our dataset. In total, our dataset provides 2609.78 km2 of labeled water surface area, covering a variety of landscapes such as rivers passing through urban regions, braided rivers in deltas, rivers passing through forests and agricultural fields, and waterbodies in plain and mountainous regions. The diversity and representativeness of our dataset make it a valuable resource for testing the limits and robustness of satellite data products and mapping methods.

All 100 labels are in the GeoTIFF format with the UInt8 data type and a single band. Each pixel can contain four possible values: 0 (no data), 1 (non-water), 2 (low-confidence water), and 3 (water). The labels are in the WGS84 (EPSG:4326) coordinate reference system. Each label has a corresponding PlanetScope image used for labeling in Labelbox. The PlanetScope images are also in the WGS84 (EPSG:4326) coordinate reference system and contain three spectral bands (red, green, and blue) in true-color composite. Based on our PS image release agreement with Planet, we converted the original surface reflectance values to byte format with possible pixel values between 0 and 255, instead of UInt16.

The label files are named using the following convention: SIDX_YYYYMMDD.tif, where “SIDX” is the unique sample ID (with X ranging from 1 to 100) and “YYYYMMDD” represents the date of the PlanetScope image used for labeling. The corresponding PlanetScope images follow the naming convention: SIDX_PSID.tif, where SIDX is the same as the label, but PSID is the original SuperDove PlanetScope image ID, allowing for the retrieval of the original surface reflectance values, provided there is access.

Our dataset is organized using the SpatioTemporal Asset Catalog (STAC) format, which is a standardized way to describe and catalog geospatial data. The STAC format provides a clear and consistent structure for storing and accessing the labels and their corresponding PlanetScope images, along with relevant metadata.

Dynamic World (DW) is a land use land cover product from Google that utilizes a deep learning model trained on their own labeled dataset. The product includes nine classes, including water, and produces a map for every Sentinel-2 image. Each Sentinel-2 image is post-processed and with clouds removed. We downloaded Sentinel-2 images within 3 d of each of the 100 labeled PlanetScope images. We also applied a not-a-number (NaN) filter, ensuring that images with at least 90 % valid pixels are considered. After applying the temporal and NaN filters, there were 53 corresponding Sentinel-2-based DW maps out of our 100 labels. From each DW map, we extracted the first band, which contains the water class. Each DW class contains continuous values between 0 and 1, where 1 denotes the highest confidence in the model prediction. We converted the continuous values to binary, thresholding at 0.3. The water class is one of the least confused classes in the DW product, so mixed pixels are less likely. Finally, we evaluated DW on our labels. Note that for evaluation we converted the low-confidence water class to water. We finally resampled the DW water class to match the resolution of the labels at 3 m using nearest-neighbor interpolation before evaluating. Note that for evaluation we merged the low-confidence water class with water. Therefore, labels were either 0 (non-water) or 1 (water).

Figure  illustrates the performance of the water class in the Dynamic World product across different biomes using IoU. IoU provides an assessment of the overlap between the predicted and ground truth water pixels, with higher values indicating better performance. The number of samples per biome varies, with some biomes having more representative data than others. For biomes with a larger number of samples, such as “Deserts and Xeric Shrublands” and “Boreal Forests and Taiga”, the IoU scores provide a more robust evaluation of the DW water class performance. Despite the variations in sample size, notable differences in performance can be observed among the biomes. It is important to note that the IoU metric is influenced by the amount of water present in each label. Higher water percentage often leads to higher IoU. However, our dataset has an average of 26.1 % surface water pixels, providing a balanced assessment of the DW water class performance.

Figure  provides a visual comparison of the Dynamic World water class predictions with the labels for two locations: Sundarban National Park, Bangladesh (SID01), and Shandong, China (SID13). The DW product appears to capture the majority of the water pixels accurately; however, it misses the narrow rivers (SID01) and it incorrectly ignores two bridges (SID13).

Table  summarizes the performance metrics for the Dynamic World water class evaluated on our hand-labeled dataset. The mean precision of 0.8812 indicates that, on average, 88.12 % of the pixels predicted as water by DW are actually water in our ground truth labels. The mean sensitivity (recall) of 0.7745 suggests that DW correctly identifies 77.45 % of the water pixels in our labels. The high mean specificity (0.9656) indicates that DW accurately classifies non-water pixels, with minimal misclassification as water. The F1 score, which is the harmonic mean of precision and recall, has a mean value of 0.7970, indicating a good balance between the two metrics. The mean IoU of 0.7216 signifies that, on average, there is a 72.16 % overlap between the predicted and ground truth water pixels. Lastly, the mean accuracy of 0.9529 shows that DW correctly classifies 95.29 % of the pixels overall, including non-water pixels. However, the high standard deviation indicates that there is a large variability in performance for almost all metrics except specificity and accuracy, since they take into account the non-water pixels.

We evaluated the performance of a deep learning model for inundation mapping that uses S1 radar imagery. This deep learning model was the competition winner at the NASA IMPACT challenge for flood detection. Unlike Dynamic World, which contained a surface water class, this method focuses on flood or more specifically inundation class. Technically, our hand-labeled dataset also labels inundation, although our labels did not focus on capturing flooding. Therefore, we are not directly comparing the S1 IMPACT flood model against the Dynamic World water class.

We processed radiometrically corrected S1 imagery from Alaska Satellite Facility (ASF)'s data repository using the Hyp3 API (application programming interface). S1 imagery was searched for each label 3 d before and after the labeled date. We clipped the S1 scenes based on the labels, and then we applied the trained model to these clipped S1 scenes using the trained model. We then evaluated the predictions from the deep learning model on our labels after resampling the imagery to match the resolution of the higher-resolution labels using nearest-neighbor interpolation. Thus, 72 S1 images were selected for this evaluation. Note that for evaluation we converted the low-confidence water class to water. Therefore, labels were either 0 (non-water) or 1 (water).

Figure  illustrates the performance of the S1-based deep learning model across different biomes using the intersection over union (IoU) metric. Performance across biomes has a large variation, with some notable differences. For example, the IMPACT model performed robustly on “Tropical and Subtropical Dry Broadleaf Forests”, “Tropical and Subtropical Moist Broadleaf Forests”, “Tundra”, and “Mangroves”. Although for “Tropical and Subtropical Coniferous Forests”, “Temperate Coniferous Forests”, and “Deserts and Xeric Shrublands”, the model performed less accurately and with large variations; this was especially true for “Mediterranean Forests, Woodlands and Scrub” where the model consistently performed poorly. The effectiveness is influenced by the fact that the training dataset of this model is focused on only five flood events globally. Therefore, performing accurately on the global surface water dataset is not the objective of this model. Nonetheless, the objective is still detection inundation, and the variation in performance provides clues to how such a model can be improved by sampling from biomes or other contexts (urban, river, lake, etc.).

Figure  provides a visual comparison of the Sentinel-1-based deep learning model's predictions with the ground truth labels for two locations: Nam Định, Vietnam (SID33), and Paymaster Landing, California, USA (SID59). The model appears to capture the majority of the water pixels accurately. However, the labels and the corresponding prediction by the S1-based model demonstrates the complexity of labeling and identifying water in a meandering braided river (SID33). In the case of SID59, the S1 model performs well, except for the coarser edges of a river in a more arid landscape.

Table  summarizes the performance metrics for the S1-based deep learning model evaluated on our hand-labeled dataset. The metrics exhibit significant variability across the evaluated labels. The S1 IMPACT model generally found it difficult to predict water pixels across several biomes. Apart from the differences in resolution, turbulent water and water located in spatially heterogeneous landscapes are more complicated to detect. Given the cloud-free observations, S1-based models can be of considerable benefit for regular monitoring and consistent observations.