Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation

• URL: http://arxiv.org/abs/2606.25128v1
• Date: Tue, 23 Jun 2026 20:02:57 GMT
• Title: Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation
• Abstract summary: We systematically evaluate Earth observation datasets alongside synthetic counterparts generated by deep generative models.<n>Our results reveal a stark misalignment: semantics-preserving perturbations such as rotation drastically alter metric scores while leaving human recognition unaffected.<n>By benchmarking semantic segmentation models trained on mixed real-synthetic datasets, we demonstrate that quality metrics rooted in ImageNet-pretrained feature spaces are unreliable indicators for geospatial data.
• Score: 4.554894288663752
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Volume and quality of datasets are crucial for deep learning model training, yet they are often constrained by availability and data acquisition costs. Synthetic data augmentation can extend existing datasets with realistic images, and the quality of these images is generally assessed through fidelity metrics such as FID, KID, IS, LPIPS and SSIM that measure structural or distributional similarity. However, such metrics, including the widely used FID, focus on visual fidelity without reflecting downstream utility, and can diverge from human perception under perturbations that are imperceptible to human observers. In this work, we systematically evaluate Earth observation datasets alongside synthetic counterparts generated by deep generative models, comparing automatic metrics against human perception and downstream tasks. Our results reveal a stark misalignment: semantics-preserving perturbations such as rotation drastically alter metric scores while leaving human recognition unaffected, and synthetic samples that score poorly on automatic metrics achieve comparable or higher perceived realism, and can improve downstream performance when combined with real data. By benchmarking semantic segmentation models trained on mixed real-synthetic datasets, we demonstrate that quality metrics rooted in ImageNet-pretrained feature spaces are unreliable indicators for geospatial data. Our findings underscore that automatic quality evaluation of synthetic datasets should be grounded in downstream task performance and human evaluation.

Related papers

• Assessment of Conditional Diffusion Model for Synthetic Histopathology Image Generation [13.18104339591425]
  We show that conventional synthetic data evaluation metrics such as Frechet Inception Distance (FID) and Inception Score (IS) may have limitations when applied to histopathology images.<n>To address these limitations, we propose for consideration modified FID and IS approaches utilizing foundation models pretrained on digital pathology datasets.
  arXiv  Detail & Related papers  (2026-08-04T17:51:40Z)
• Benchmarking Face Recognition without Real Faces [10.351122504676573]
  Synthetic face datasets have become effective enough to train face recognition models with accuracy rivaling that of models trained on real photographs.<n>This progress sidesteps the ethical and legal burdens of collecting real biometric data, yet evaluation has not kept pace.<n>We ask whether synthetic datasets can replace real benchmarks for face recognition evaluation.
  arXiv  Detail & Related papers  (2026-07-16T12:47:24Z)
• Training-Free Metrics for Synthetic Object Detection Data: A Proxy for Detector Performance [3.052250424013348]
  Conditional-Composition Domain Match (CCDM) serves as a proxy for the relative utility of candidate synthetic training sets for downstream detection.<n>Experiments on the VisDrone-DET dataset show that the CCDM metric families achieve a Spearman correlation of 1.0 with the downstream performance of YOLOv8.
  arXiv  Detail & Related papers  (2026-06-18T05:46:23Z)
• Discriminative Span as a Predictor of Synthetic Data Utility via Classifier Reconstruction [0.552480439325792]
  A widely adopted solution is to generate synthetic positive data using image-to-image transformations applied to negative samples.<n>We propose a geometry-driven metric that predicts the utility of synthetic data without requiring model training.<n>We show that this metric exhibits strong correlation with downstream classification performance of CNNs trained on mixtures of real negative and synthetic positive data.
  arXiv  Detail & Related papers  (2026-05-10T18:32:31Z)
• Toward Visually Realistic Simulation: A Benchmark for Evaluating Robot Manipulation in Simulation [62.51953630639423]
  Existing benchmarks lack visual realism, creating a large domain gap between simulation and reality.<n>We propose VISER, a visually realistic benchmark for evaluating robot manipulation in simulation.<n>VISER features a high-fidelity dataset of over 1,000 3D assets with physically-based rendering (PBR) materials, along with 3D scenes created from these assets through curated layouts or generation.
  arXiv  Detail & Related papers  (2026-05-07T14:13:05Z)
• A Style-Based Profiling Framework for Quantifying the Synthetic-to-Real Gap in Autonomous Driving Datasets [9.788200709163064]
  This paper introduces a profile extraction and discovery framework for characterizing the style profiles underlying both synthetic and real image datasets.<n>Our framework combines Gram matrix-based style extraction with metric learning optimized for intra-class compactness and inter-class separation to extract style embeddings.
  arXiv  Detail & Related papers  (2025-10-11T13:09:41Z)
• Explainable Human-in-the-Loop Segmentation via Critic Feedback Signals [0.20999222360659608]
  We propose a human-in-the-loop interactive framework that enables interventional learning through targeted human corrections of segmentation outputs.<n>We demonstrate that our framework improves segmentation accuracy by up to 9 mIoU points on challenging cubemap data.<n>This work provides a practical framework for researchers and practitioners seeking to build segmentation systems that are accurate, robust to dataset biases, data-efficient, and adaptable to real-world domains such as urban climate monitoring and autonomous driving.
  arXiv  Detail & Related papers  (2025-10-11T01:16:41Z)
• Understanding the Influence of Synthetic Data for Text Embedders [52.04771455432998]
  We first reproduce and publicly release the synthetic data proposed by Wang et al.<n>We critically examine where exactly synthetic data improves model generalization.<n>Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders.
  arXiv  Detail & Related papers  (2025-09-07T19:28:52Z)
• RoHOI: Robustness Benchmark for Human-Object Interaction Detection [84.78366452133514]
  Human-Object Interaction (HOI) detection is crucial for robot-human assistance, enabling context-aware support.<n>We introduce the first benchmark for HOI detection, evaluating model resilience under diverse challenges.<n>Our benchmark, RoHOI, includes 20 corruption types based on the HICO-DET and V-COCO datasets and a new robustness-focused metric.
  arXiv  Detail & Related papers  (2025-07-12T01:58:04Z)
• Reliability in Semantic Segmentation: Can We Use Synthetic Data? [69.28268603137546]
  We show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. This synthetic data is employed to evaluate the robustness of pretrained segmenters. We demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters.
  arXiv  Detail & Related papers  (2023-12-14T18:56:07Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.