Exploring Generative AI for Sim2Real in Driving Data Synthesis

• URL: http://arxiv.org/abs/2404.09111v1
• Date: Sun, 14 Apr 2024 01:23:19 GMT
• Title: Exploring Generative AI for Sim2Real in Driving Data Synthesis
• Authors: Haonan Zhao, Yiting Wang, Thomas Bashford-Rogers, Valentina Donzella, Kurt Debattista,
• Abstract summary: Driving simulators offer a solution by automatically generating various driving scenarios with corresponding annotations, but the simulation-to-reality (Sim2Real) domain gap remains a challenge. This paper applied three different generative AI methods to leverage semantic label maps from a driving simulator as a bridge for the creation of realistic datasets. Experiments show that although GAN-based methods are adept at generating high-quality images when provided with manually annotated labels, ControlNet produces synthetic datasets with fewer artefacts and more structural fidelity when using simulator-generated labels.
• Score: 6.769182994217369
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Datasets are essential for training and testing vehicle perception algorithms. However, the collection and annotation of real-world images is time-consuming and expensive. Driving simulators offer a solution by automatically generating various driving scenarios with corresponding annotations, but the simulation-to-reality (Sim2Real) domain gap remains a challenge. While most of the Generative Artificial Intelligence (AI) follows the de facto Generative Adversarial Nets (GANs)-based methods, the recent emerging diffusion probabilistic models have not been fully explored in mitigating Sim2Real challenges for driving data synthesis. To explore the performance, this paper applied three different generative AI methods to leverage semantic label maps from a driving simulator as a bridge for the creation of realistic datasets. A comparative analysis of these methods is presented from the perspective of image quality and perception. New synthetic datasets, which include driving images and auto-generated high-quality annotations, are produced with low costs and high scene variability. The experimental results show that although GAN-based methods are adept at generating high-quality images when provided with manually annotated labels, ControlNet produces synthetic datasets with fewer artefacts and more structural fidelity when using simulator-generated labels. This suggests that the diffusion-based approach may provide improved stability and an alternative method for addressing Sim2Real challenges.

Related papers

• CARLA2Real: a tool for reducing the sim2real gap in CARLA simulator [2.8978140690127328]
  We employ a state-of-the-art approach to enhance the photorealism of simulated data, aligning them with the visual characteristics of real-world datasets. Based on this, we developed CARLA2Real, an easy-to-use, publicly available tool (plug-in) for the widely used and open-source CARLA simulator. This tool enhances the output of CARLA in near real-time, achieving a frame rate of 13 FPS, translating it to the visual style and realism of real-world datasets.
  arXiv  Detail & Related papers  (2024-10-23T19:33:30Z)
• Automatic AI Model Selection for Wireless Systems: Online Learning via Digital Twinning [50.332027356848094]
  AI-based applications are deployed at intelligent controllers to carry out functionalities like scheduling or power control. The mapping between context and AI model parameters is ideally done in a zero-shot fashion. This paper introduces a general methodology for the online optimization of AMS mappings.
  arXiv  Detail & Related papers  (2024-06-22T11:17:50Z)
• SimGen: Simulator-conditioned Driving Scene Generation [50.03358485083602]
  We introduce a simulator-conditioned scene generation framework called SimGen. SimGen learns to generate diverse driving scenes by mixing data from the simulator and the real world. It achieves superior generation quality and diversity while preserving controllability based on the text prompt and the layout pulled from a simulator.
  arXiv  Detail & Related papers  (2024-06-13T17:58:32Z)
• Are NeRFs ready for autonomous driving? Towards closing the real-to-simulation gap [6.393953433174051]
  We propose a novel perspective for addressing the real-to-simulated data gap. We conduct the first large-scale investigation into the real-to-simulated data gap in an autonomous driving setting. Our results show notable improvements in model robustness to simulated data, even improving real-world performance in some cases.
  arXiv  Detail & Related papers  (2024-03-24T11:09:41Z)
• Augmented Reality based Simulated Data (ARSim) with multi-view consistency for AV perception networks [47.07188762367792]
  We present ARSim, a framework designed to enhance real multi-view image data with 3D synthetic objects of interest. We construct a simplified virtual scene using real data and strategically place 3D synthetic assets within it. The resulting augmented multi-view consistent dataset is used to train a multi-camera perception network for autonomous vehicles.
  arXiv  Detail & Related papers  (2024-03-22T17:49:11Z)
• BITS: Bi-level Imitation for Traffic Simulation [38.28736985320897]
  We take a data-driven approach and propose a method that can learn to generate traffic behaviors from real-world driving logs. We empirically validate our method, named Bi-level Imitation for Traffic Simulation (BITS), with scenarios from two large-scale driving datasets. As part of our core contributions, we develop and open source a software tool that unifies data formats across different driving datasets.
  arXiv  Detail & Related papers  (2022-08-26T02:17:54Z)
• TRoVE: Transforming Road Scene Datasets into Photorealistic Virtual Environments [84.6017003787244]
  This work proposes a synthetic data generation pipeline to address the difficulties and domain-gaps present in simulated datasets. We show that using annotations and visual cues from existing datasets, we can facilitate automated multi-modal data generation.
  arXiv  Detail & Related papers  (2022-08-16T20:46:08Z)
• Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation [98.51313127382937]
  We focus on the use of labels in the synthetic domain alone. Our approach introduces both a way to learn neural-invariant representations and a theoretically inspired view on how to sample the data from the simulator. We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data.
  arXiv  Detail & Related papers  (2021-11-15T18:37:43Z)
• Unlimited Resolution Image Generation with R2D2-GANs [69.90258455164513]
  We present a novel simulation technique for generating high quality images of any predefined resolution. This method can be used to synthesize sonar scans of size equivalent to those collected during a full-length mission. The data produced is continuous, realistically-looking, and can also be generated at least two times faster than the real speed of acquisition.
  arXiv  Detail & Related papers  (2020-03-02T17:49:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.