Grounding Synthetic Data Generation With Vision and Language Models

• URL: http://arxiv.org/abs/2603.09625v1
• Date: Tue, 10 Mar 2026 13:03:53 GMT
• Title: Grounding Synthetic Data Generation With Vision and Language Models
• Abstract summary: We propose a vision-language grounded framework for interpretable synthetic data augmentation and evaluation in remote sensing.<n>Based on this framework, we introduce ARAS400k: A large-scale Remote sensing dataset Augmented with Synthetic data for segmentation and captioning.<n>ARAS400k enables the automated evaluation of synthetic data by analyzing semantic composition, minimizing caption redundancy, and verifying cross-modal consistency between visual structures and language descriptions.
• Score: 4.554894288663752
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning models benefit from increasing data diversity and volume, motivating synthetic data augmentation to improve existing datasets. However, existing evaluation metrics for synthetic data typically calculate latent feature similarity, which is difficult to interpret and does not always correlate with the contribution to downstream tasks. We propose a vision-language grounded framework for interpretable synthetic data augmentation and evaluation in remote sensing. Our approach combines generative models, semantic segmentation and image captioning with vision and language models. Based on this framework, we introduce ARAS400k: A large-scale Remote sensing dataset Augmented with Synthetic data for segmentation and captioning, containing 100k real images and 300k synthetic images, each paired with segmentation maps and descriptions. ARAS400k enables the automated evaluation of synthetic data by analyzing semantic composition, minimizing caption redundancy, and verifying cross-modal consistency between visual structures and language descriptions. Experimental results indicate that while models trained exclusively on synthetic data reach competitive performance levels, those trained with augmented data (a combination of real and synthetic images) consistently outperform real-data baselines. Consequently, this work establishes a scalable benchmark for remote sensing tasks, specifically in semantic segmentation and image captioning. The dataset is available at zenodo.org/records/18890661 and the code base at github.com/caglarmert/ARAS400k.

Related papers

• A Framework for Low-Effort Training Data Generation for Urban Semantic Segmentation [15.541453405140485]
  Synthetic datasets are widely used for training urban scene recognition models, but even highly realistic renderings show a noticeable gap to real imagery.<n>We present a new framework that adapts an off-the-shelf diffusion model to a target domain using only imperfect pseudo-labels.<n>It generates high-fidelity, target-aligned images from semantic maps of any synthetic dataset, including low-effort sources created in hours rather than months.
  arXiv  Detail & Related papers  (2025-10-13T16:12:29Z)
• Scaling Laws of Synthetic Data for Language Models [125.41600201811417]
  We introduce SynthLLM, a scalable framework that transforms pre-training corpora into diverse, high-quality synthetic datasets.<n>Our approach achieves this by automatically extracting and recombining high-level concepts across multiple documents using a graph algorithm.
  arXiv  Detail & Related papers  (2025-03-25T11:07:12Z)
• Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation [79.71072337496351]
  CoSyn is a framework that creates synthetic text-rich multimodal data.<n>It can generate high-quality instruction-tuning data.<n>It can also produce synthetic pointing data, enabling vision-language models to ground information within input images.
  arXiv  Detail & Related papers  (2025-02-20T18:55:30Z)
• RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm [34.02250139766494]
  Contrastive Language-Image Pre-training (CLIP) demonstrates promising performance on a variety of benchmarks.<n>A substantial volume of multimodal interleaved documents remains underutilized for contrastive vision-language representation learning.<n>We establish a Real-World Data Extraction pipeline to extract high-quality images and texts.<n>Then we design a hierarchical retrieval method to efficiently associate each image with multiple semantically relevant realistic texts.<n>We construct RealSyn, a dataset combining realistic and synthetic texts, available in three scales: 15M, 30M, and 100M.
  arXiv  Detail & Related papers  (2025-02-18T03:58:38Z)
• SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data [78.70620682374624]
  We introduce SynFER, a novel framework for synthesizing facial expression image data based on high-level textual descriptions.<n>To ensure the quality and reliability of the synthetic data, we propose a semantic guidance technique and a pseudo-label generator.<n>Results validate the efficacy of our approach and the synthetic data.
  arXiv  Detail & Related papers  (2024-10-13T14:58:21Z)
• Learning from Synthetic Data for Visual Grounding [55.21937116752679]
  We show that SynGround can improve the localization capabilities of off-the-shelf vision-and-language models.<n>Data generated with SynGround improves the pointing game accuracy of a pretrained ALBEF and BLIP models by 4.81% and 17.11% absolute percentage points, respectively.
  arXiv  Detail & Related papers  (2024-03-20T17:59:43Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Adapting to Unseen Vendor Domains for MRI Lesion Segmentation [0.08156494881838945]
  We investigate an unsupervised image translation model to augment MR images from a source dataset to a target dataset. We consider three configurations of augmentation between datasets consisting of translation between images, between scanner vendors, and from labels to images. It was found that the segmentation models trained on synthetic data from labels to images configuration yielded the closest performance to the segmentation model trained directly on the target dataset.
  arXiv  Detail & Related papers  (2021-08-14T01:09:43Z)
• PennSyn2Real: Training Object Recognition Models without Human Labeling [12.923677573437699]
  We propose PennSyn2Real - a synthetic dataset consisting of more than 100,000 4K images of more than 20 types of micro aerial vehicles (MAVs) The dataset can be used to generate arbitrary numbers of training images for high-level computer vision tasks such as MAV detection and classification. We show that synthetic data generated using this framework can be directly used to train CNN models for common object recognition tasks such as detection and segmentation.
  arXiv  Detail & Related papers  (2020-09-22T02:53:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.