Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation

• URL: http://arxiv.org/abs/2607.04770v1
• Date: Mon, 06 Jul 2026 08:03:31 GMT
• Title: Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation
• Abstract summary: Cam2Sim is a tool that transforms real-world driving recordings into playable CARLA simulation scenarios.<n>The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation.<n>We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model.
• Score: 0.9114245816258512
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Simulation-based testing enables safe and repeatable evaluation of autonomous driving systems, but its effectiveness is limited by the gap between synthetic simulator outputs and real-world camera observations. To address this problem, we present Cam2Sim, a tool that transforms real-world driving recordings into playable CARLA simulation scenarios. Starting from camera images and poses, Cam2Sim reconstructs road geometry, ego trajectories, parked vehicles, and simulation assets, and augments the reconstructed environment with Gaussian Splatting to render camera observations that resemble the original recording. The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation, CARLA scenario construction, Gaussian Splatting training, trajectory replay, and closed-loop execution with a system under test. We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model, comparing reconstruction quality, image-generation quality, and closed-loop behavior against both a simulation-only baseline and the real-world target. Results show that Gaussian-Splatting-based rendering reduces the visual gap with respect to standard simulator rendering and improves behavioral similarity to the real-world reference runs. The artifact is publicly available at https: //github.com/ast-fortiss-tum/cam2sim, and a screencast showing the tool is available at https://youtu.be/KmZ74l1__lI

Related papers

• Point as Skeleton: Accumulated Point Cloud Enhanced Autoregressive Generation for Closed-Loop Autonomous Driving Simulation [58.86413499077093]
  We present textbfemphPoint as Skeleton, a generative sensor simulation framework for state-updated autoregressive driving video generation.<n>To support closed-loop rollout, we introduce Reset-and-Roll, which adapts rolling diffusion inference to simulation.<n>Experiments on nuScenes and nuPlan show that textitPoint as Skeleton improves autoregressive generation quality during closed-loop rollout.
  arXiv  Detail & Related papers  (2026-07-07T17:20:51Z)
• ReconDreamer-RL: Enhancing Reinforcement Learning via Diffusion-based Scene Reconstruction [26.402373173809753]
  ReconDreamer-RL is a framework designed to integrate video diffusion priors into scene reconstruction to aid reinforcement learning.<n>We show that ReconDreamer-RL improves end-to-end autonomous driving training, outperforming imitation learning methods with a 5x reduction in the Collision Ratio.
  arXiv  Detail & Related papers  (2025-08-11T16:45:55Z)
• Bench2Drive-R: Turning Real World Data into Reactive Closed-Loop Autonomous Driving Benchmark by Generative Model [63.336123527432136]
  We introduce Bench2Drive-R, a generative framework that enables reactive closed-loop evaluation.<n>Unlike existing video generative models for autonomous driving, the proposed designs are tailored for interactive simulation.<n>We compare the generation quality of Bench2Drive-R with existing generative models and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-11T06:35:18Z)
• HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for Autonomous Driving [48.84595398410479]
  HUGSIM is a closed-loop, photo-realistic, and real-time simulator for evaluating autonomous driving algorithms.<n>We tackle challenges of novel view synthesis in closed-loop scenarios, including viewpoint extrapolation and 360-degree vehicle rendering.<n> HUGSIM offers a comprehensive benchmark across more than 70 sequences from KITTI-360, nuScenes, and PandaSet, along with over 400 varying scenarios.
  arXiv  Detail & Related papers  (2024-12-02T17:07:59Z)
• CARLA2Real: a tool for reducing the sim2real appearance gap in CARLA simulator [2.478819644330144]
  We employ a state-of-the-art approach to enhance the photorealism of simulated data, aligning them with the visual characteristics of real-world datasets.<n>Based on this, we developed CARLA2Real, an easy-to-use, publicly available tool (plug-in) for the widely used and open-source CARLA simulator.<n>This tool enhances the output of CARLA in near real-time, achieving a frame rate of 13 FPS, translating it to the visual style and realism of real-world datasets.
  arXiv  Detail & Related papers  (2024-10-23T19:33:30Z)
• Deep Reinforcement Learning for Adverse Garage Scenario Generation [5.482809279542029]
  This thesis proposes an automated program generation framework for autonomous driving simulation testing. Based on deep reinforcement learning, this framework can generate different 2D ground script codes, on which 3D model files and map model files are built. The generated 3D ground scenes are displayed in the Carla simulator, where experimenters can use this scene for navigation algorithm simulation testing.
  arXiv  Detail & Related papers  (2024-07-01T14:41:18Z)
• XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis [84.23233209017192]
  This paper presents a synthetic dataset for novel driving view synthesis evaluation.<n>It includes testing images captured by deviating from the training trajectory by $1-4$ meters.<n>We establish the first realistic benchmark for evaluating existing NVS approaches under front-only and multicamera settings.
  arXiv  Detail & Related papers  (2024-06-26T14:00:21Z)
• UniSim: A Neural Closed-Loop Sensor Simulator [76.79818601389992]
  We present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene. We incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions.
  arXiv  Detail & Related papers  (2023-08-03T17:56:06Z)
• Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data [56.49494318285391]
  We present Sim2Seg, a re-imagining of RCAN that crosses the visual reality gap for off-road autonomous driving. This is done by learning to translate randomized simulation images into simulated segmentation and depth maps. This allows us to train an end-to-end RL policy in simulation, and directly deploy in the real-world.
  arXiv  Detail & Related papers  (2022-10-25T17:50:36Z)
• SimNet: Learning Reactive Self-driving Simulations from Real-world Observations [10.035169936164504]
  We present an end-to-end trainable machine learning system capable of realistically simulating driving experiences. This can be used for the verification of self-driving system performance without relying on expensive and time-consuming road testing.
  arXiv  Detail & Related papers  (2021-05-26T05:14:23Z)
• DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
  We introduce a novel high-quality neural simulator referred to as DriveGAN. DriveGAN achieves controllability by disentangling different components without supervision. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
  arXiv  Detail & Related papers  (2021-04-30T15:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.