Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents

• URL: http://arxiv.org/abs/2402.05746v3
• Date: Wed, 26 Jun 2024 10:44:58 GMT
• Title: Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents
• Authors: Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, Yanfeng Wang,
• Abstract summary: ChatSim is the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. ChatSim employs a novel multi-camera neural field method to generate photo-realistic outcomes.
• Score: 49.11254369459406
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene simulation in autonomous driving has gained significant attention because of its huge potential for generating customized data. However, existing editable scene simulation approaches face limitations in terms of user interaction efficiency, multi-camera photo-realistic rendering and external digital assets integration. To address these challenges, this paper introduces ChatSim, the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. To enable editing with high command flexibility,~ChatSim leverages a large language model (LLM) agent collaboration framework. To generate photo-realistic outcomes, ChatSim employs a novel multi-camera neural radiance field method. Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering. Our experiments on Waymo Open Dataset demonstrate that ChatSim can handle complex language commands and generate corresponding photo-realistic scene videos.

Related papers

• SimGen: Simulator-conditioned Driving Scene Generation [50.03358485083602]
  We introduce a simulator-conditioned scene generation framework called SimGen. SimGen learns to generate diverse driving scenes by mixing data from the simulator and the real world. It achieves superior generation quality and diversity while preserving controllability based on the text prompt and the layout pulled from a simulator.
  arXiv  Detail & Related papers  (2024-06-13T17:58:32Z)
• URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images [39.0780707100513]
  We present an integrated end-to-end pipeline that generates simulation scenes complete with articulated kinematic and dynamic structures from real-world images. In doing so, our work provides both a pipeline for large-scale generation of simulation environments and an integrated system for training robust robotic control policies.
  arXiv  Detail & Related papers  (2024-05-19T20:01:29Z)
• S-NeRF++: Autonomous Driving Simulation via Neural Reconstruction and Generation [21.501865765631123]
  S-NeRF++ is an innovative autonomous driving simulation system based on neural reconstruction. S-NeRF++ is trained on widely-used self-driving datasets such as nuScenes and radiance. System effectively utilizes noisy and sparse LiDAR data to refine training and address depth outliers.
  arXiv  Detail & Related papers  (2024-02-03T10:35:42Z)
• GarchingSim: An Autonomous Driving Simulator with Photorealistic Scenes and Minimalist Workflow [24.789118651720045]
  We introduce an autonomous driving simulator with photorealistic scenes. The simulator is able to communicate with external algorithms through ROS2 or Socket.IO. We implement a highly accurate vehicle dynamics model within the simulator to enhance the realism of the vehicle's physical effects.
  arXiv  Detail & Related papers  (2024-01-28T23:26:15Z)
• Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data [56.49494318285391]
  We present Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world.
  arXiv  Detail & Related papers  (2022-10-25T17:50:36Z)
• VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles [131.2240621036954]
  We present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras. We demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle.
  arXiv  Detail & Related papers  (2021-11-23T18:58:10Z)
• DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
  We introduce a novel high-quality neural simulator referred to as DriveGAN. DriveGAN achieves controllability by disentangling different components without supervision. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
  arXiv  Detail & Related papers  (2021-04-30T15:30:05Z)
• GeoSim: Photorealistic Image Simulation with Geometry-Aware Composition [81.24107630746508]
  We present GeoSim, a geometry-aware image composition process that synthesizes novel urban driving scenes. We first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. The resulting synthetic images are photorealistic, traffic-aware, and geometrically consistent, allowing image simulation to scale to complex use cases.
  arXiv  Detail & Related papers  (2021-01-16T23:00:33Z)
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving [27.948417322786575]
  We present a simple yet effective approach to generate realistic scenario sensor data. Our approach uses texture-mapped surfels to efficiently reconstruct the scene from an initial vehicle pass or set of passes. We then leverage a SurfelGAN network to reconstruct realistic camera images for novel positions and orientations of the self-driving vehicle.
  arXiv  Detail & Related papers  (2020-05-08T04:01:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.