Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI

• URL: http://arxiv.org/abs/2607.12874v1
• Date: Tue, 14 Jul 2026 15:22:26 GMT
• Title: Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI
• Abstract summary: Deep learning vision for scientific applications requires collecting and annotating large datasets in a laborious, expensive and error-prone process.<n>We present GraNatPy, Python package with metrics to guide improvement of rendered scene.<n>We show that increase in realism, diversity and size of rendered dataset correlates with improved visual perception of the scene and higher zero-shot performance of an object detection model.
• Score: 3.1193133788500824
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning computer vision for scientific applications requires collecting and annotating large datasets in a laborious, expensive and error-prone process. Synthetic data generation through 3D modelling and rendering may simplify this process and increase the accuracy of annotations by generating them programmatically. However, minimising the domain gap between real and synthetic images visually is subjective and lacks systematic quantitative guidance. We present GraNatPy, a Python package with metrics to guide improvement of the rendered scene. We show that quantifiable increase in realism, diversity and size of rendered dataset correlates with improved visual perception of the scene and higher zero-shot performance of an object detection model. Furthermore, we demonstrated using photographs of virological plaque assays that gradient similarity affects performance on small object detection, which can be improved by mixing real and synthetic data. Finally, we turn procedural data rendering into an agentic skill (SynthClaw) to automate the procedural parameter optimisation.

Related papers

• Synthetic Dataset Generation and Validation for Robotic Surgery Instrument Segmentation [4.731220357458455]
  A 3D reconstruction of the Da Vinci robotic arms was refined and animated in Autodesk Maya.<n>To validate the realism and effectiveness of the generated data, several segmentation models were trained under controlled ratios of real and synthetic data.
  arXiv  Detail & Related papers  (2026-02-14T18:29:03Z)
• Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data [53.040873127309766]
  We propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning.<n>Our method outperforms existing models on both in-dataset and cross-dataset evaluations.
  arXiv  Detail & Related papers  (2025-09-08T17:58:06Z)
• Synthetic Dataset Generation for Autonomous Mobile Robots Using 3D Gaussian Splatting for Vision Training [0.708987965338602]
  We propose a novel method for automatically generating annotated synthetic data in Unreal Engine.<n>We demonstrate that synthetic datasets can achieve performance comparable to that of real-world datasets.<n>This is the first application of synthetic data for training object detection algorithms in robot soccer.
  arXiv  Detail & Related papers  (2025-06-05T14:37:40Z)
• Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles [81.29018359825872]
  This paper consolidates a set of good practices to finetune large pretrained models for a real-world task.<n>Specifically, we develop several strategies to account for discrepancies between the synthetic data and real driving data.<n>Our insights lead to effective finetuning that results in a $68.8%$ reduction in FID for novel view synthesis over prior arts.
  arXiv  Detail & Related papers  (2024-12-19T03:39:13Z)
• Improving Object Detection by Modifying Synthetic Data with Explainable AI [3.0519884745675485]
  We propose a novel conceptual approach to improve the efficiency of designing synthetic images.<n>XAI techniques guide a human-in-the-loop process of modifying 3D mesh models used to generate these images.<n>We show that synthetic data can improve detection of vehicles in orientations unseen in training by 4.6%.
  arXiv  Detail & Related papers  (2024-12-02T13:24:43Z)
• Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization [62.157627519792946]
  We introduce a novel framework called bridged transfer, which initially employs synthetic images for fine-tuning a pre-trained model to improve its transferability. We propose dataset style inversion strategy to improve the stylistic alignment between synthetic and real images. Our proposed methods are evaluated across 10 different datasets and 5 distinct models, demonstrating consistent improvements.
  arXiv  Detail & Related papers  (2024-03-28T22:25:05Z)
• Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems [80.62854148838359]
  Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. We use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data. Our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.
  arXiv  Detail & Related papers  (2024-03-23T22:32:06Z)
• A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation [42.2398858786125]
  Deep learning in computer vision has achieved great success with the price of large-scale labeled training data. The uncontrollable data collection process produces non-IID training and test data, where undesired duplication may exist. To circumvent them, an alternative is to generate synthetic data via 3D rendering with domain randomization.
  arXiv  Detail & Related papers  (2023-03-16T09:03:52Z)
• Is synthetic data from generative models ready for image recognition? [69.42645602062024]
  We study whether and how synthetic images generated from state-of-the-art text-to-image generation models can be used for image recognition tasks. We showcase the powerfulness and shortcomings of synthetic data from existing generative models, and propose strategies for better applying synthetic data for recognition tasks.
  arXiv  Detail & Related papers  (2022-10-14T06:54:24Z)
• Intrinsic Autoencoders for Joint Neural Rendering and Intrinsic Image Decomposition [67.9464567157846]
  We propose an autoencoder for joint generation of realistic images from synthetic 3D models while simultaneously decomposing real images into their intrinsic shape and appearance properties. Our experiments confirm that a joint treatment of rendering and decomposition is indeed beneficial and that our approach outperforms state-of-the-art image-to-image translation baselines both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-06-29T12:53:58Z)
• Can Synthetic Data Improve Object Detection Results for Remote Sensing Images? [15.466412729455874]
  We propose the use of realistic synthetic data with a wide distribution to improve the performance of remote sensing image aircraft detection. We randomly set the parameters during rendering, such as the size of the instance and the class of background images. In order to make the synthetic images more realistic, we refine the synthetic images at the pixel level using CycleGAN with real unlabeled images.
  arXiv  Detail & Related papers  (2020-06-09T02:23:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.