Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos

• URL: http://arxiv.org/abs/2606.07366v1
• Date: Fri, 05 Jun 2026 15:09:27 GMT
• Title: Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos
• Abstract summary: Dash2Sim is a framework that turns in-the-wild dashcam videos into metric, geo-referenced 4D driving logs compatible with existing simulators.<n>We show that dense depth recovered by Dash2Sim improves novel-view synthesis quality by up to 19% on perceptual metrics.
• Score: 51.46430520173442
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-driving simulations typically rely on data collected in a small number of cities or on hand-authored synthetic scenarios. Dashcam videos cover a far broader range of locations and situations, including rare or long-tailed scenarios. They are considered less usable for simulation because it is difficult to recover accurate 4D scenes from monocular in-the-wild videos. Work zones are one such class of long-tailed situations that dashcams capture. We present Dash2Sim, a framework that turns in-the-wild monocular dashcam videos into metric, geo-referenced 4D driving logs compatible with existing simulators, and verifies eachone against an independently maintained map without annotations. We apply Dash2Sim to a large video corpus to create the ROADWork4D benchmark dataset, which spans 4,244 scenes with 2.7M 3D objects across 17 cities. On a verified subset ROADWork4D-CL (2,201 scenes), we study privileged closed-loop planners and find that work zone scenarios are difficult: while rule-based and hybrid planners generalize better than learning-based ones, all fall short, failing to make the lane changes that temporary work zone channels require. Beyond planning, dense depth recovered by Dash2Sim improves novel-view synthesis quality by up to 19% on perceptual metrics, suggesting its potential to provide rich conditioning for closed-loop sensor simulation from monocular videos.

Related papers

• Cam2Sim: Neural Scenario Reconstruction for Closed-Loop Autonomous Driving Simulation [0.9114245816258512]
  Cam2Sim is a tool that transforms real-world driving recordings into playable CARLA simulation scenarios.<n>The framework supports ROS-based data extraction, parked-vehicle detection, OpenStreetMap-based map generation.<n>We validate Cam2Sim on a real-world urban-driving scenario with a camera-based end-to-end driving model.
  arXiv  Detail & Related papers  (2026-07-06T08:03:31Z)
• Evaluating Video Models as Simulators of Multi-Person Pedestrian Trajectories [1.2676356746752893]
  We benchmark text-to-video (T2V) and image-to-video (I2V) models as implicit simulators of pedestrian dynamics.<n>A key component is a method to reconstruct 2D bird's-eye view trajectories from pixel-space without known camera parameters.
  arXiv  Detail & Related papers  (2025-10-23T04:06:58Z)
• Multi-View 3D Point Tracking [67.21282192436031]
  We introduce the first data-driven multi-view 3D point tracker, designed to track arbitrary points in dynamic scenes using multiple camera views.<n>Our model directly predicts 3D correspondences using a practical number of cameras.<n>We train on 5K synthetic multi-view Kubric sequences and evaluate on two real-world benchmarks.
  arXiv  Detail & Related papers  (2025-08-28T17:58:20Z)
• RoundaboutHD: High-Resolution Real-World Urban Environment Benchmark for Multi-Camera Vehicle Tracking [2.1947594726633324]
  RoundaboutHD is a high-resolution multi-camera vehicle tracking benchmark dataset.<n>It offers temporal consistency video footage and enhanced challenges.<n>We provide baseline results for vehicle detection, single-camera tracking, image-based vehicle re-identification, and multi-camera tracking.
  arXiv  Detail & Related papers  (2025-07-11T16:30:27Z)
• DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving [83.27075316161086]
  Photorealistic 4D reconstruction of street scenes is essential for developing real-world simulators in autonomous driving.<n>We introduce the Large 4D Gaussian Reconstruction Model (DrivingRecon), a generalizable driving scene reconstruction model.<n>We show that DrivingRecon significantly improves scene reconstruction quality and novel view synthesis compared to existing methods.
  arXiv  Detail & Related papers  (2024-12-12T08:10:31Z)
• Stag-1: Towards Realistic 4D Driving Simulation with Video Generation Model [83.31688383891871]
  We propose a Spatial-Temporal simulAtion for drivinG (Stag-1) model to reconstruct real-world scenes.<n>Stag-1 constructs continuous 4D point cloud scenes using surround-view data from autonomous vehicles.<n>It decouples spatial-temporal relationships and produces coherent driving videos.
  arXiv  Detail & Related papers  (2024-12-06T18:59:56Z)
• DrivingSphere: Building a High-fidelity 4D World for Closed-loop Simulation [54.02069690134526]
  We propose DrivingSphere, a realistic and closed-loop simulation framework. Its core idea is to build 4D world representation and generate real-life and controllable driving scenarios. By providing a dynamic and realistic simulation environment, DrivingSphere enables comprehensive testing and validation of autonomous driving algorithms.
  arXiv  Detail & Related papers  (2024-11-18T03:00:33Z)
• UniSim: A Neural Closed-Loop Sensor Simulator [76.79818601389992]
  We present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene. We incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions.
  arXiv  Detail & Related papers  (2023-08-03T17:56:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.