SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics

• URL: http://arxiv.org/abs/2605.08528v1
• Date: Fri, 08 May 2026 22:23:11 GMT
• Title: SceneFactory: GPU-Accelerated Multi-Agent Driving Simulation with Physics-Based Vehicle Dynamics
• Abstract summary: SceneFactory is a GPU-vectorized platform for procedural scene construction, physics-based multi-agent simulation, and RL in autonomous-driving environments.<n> SceneFactory converts Open Motion road topologies into simulation-ready USD worlds, runs many worlds concurrently on one GPU, populates each with multiple articulated PhysX vehicles, and maps precipitation and road-surface type to PhysX material friction coefficients.
• Score: 8.411603210370858
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous-driving simulators typically trade physical fidelity for scalable parallelism. Physics-based platforms such as CARLA and MetaDrive provide articulated vehicle dynamics and contact, but their non-vectorized interfaces make batched training difficult. GPU-batched systems such as Waymax and GPUDrive scale to hundreds of scenarios by replacing rigid-body physics with simplified kinematic models, omitting tire--road interaction, suspension, contact dynamics, and road-condition-dependent friction. We introduce SceneFactory, a GPU-vectorized platform for procedural scene construction, physics-based multi-agent simulation, and RL in autonomous-driving environments. Built on NVIDIA Isaac Sim + Isaac Lab, SceneFactory represents worlds and agents as batched tensors: control, observations, rewards, resets, and policy inference run as GPU tensor operations over the Isaac Lab tensor API. SceneFactory converts Waymo Open Motion Dataset road topologies into simulation-ready USD worlds, runs many worlds concurrently on one GPU, populates each with multiple articulated PhysX vehicles, and maps precipitation and road-surface type to PhysX material friction coefficients. With GPU vectorization, SceneFactory achieves up to 127$\times$ higher throughput than a non-vectorized PhysX baseline on the same GPU and physics solver, reaching 19,250 controlled-agent simulation steps per second at 256 worlds $\times$ 16 agents. Cross-simulator transfer reveals an asymmetric dynamics gap: physics-grounded RL policies transfer to a simplified kinematic bicycle model with 99.5% success, whereas reverse transfer drops to 47.3%. Under wet-road friction, friction-aware policies reduce mean peak DRAC from 58.7 to 27.8,m/s$^2$ without sacrificing goal reach. SceneFactory shows that scalable autonomous-driving training need not discard articulated rigid-body dynamics or physically grounded road-condition variation.

Related papers

• PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control [85.54736105647055]
  PhysStream is an autoregressive model for physics-grounded image-to-video synthesis.<n>It supports fine-grained motion control via sparse velocity-increment signals.<n>PhysStream is preferred by human evaluators in over 85% of in-the-wild comparisons.
  arXiv  Detail & Related papers  (2026-09-15T17:55:13Z)
• TerraZero: Procedural Driving Simulation for Zero-Demonstration Self-Play at Scale [18.077282979391423]
  TerraZero is a procedural driving simulator and self-play training stack.<n>A C engine runs simulation on the CPU and policy inference on the GPU over a zero-copy path.<n>Every reported policy trains from scratch by reinforcement learning alone on a compute-efficient self-play recipe.
  arXiv  Detail & Related papers  (2026-07-14T17:59:02Z)
• GS-Playground: A High-Throughput Photorealistic Simulator for Vision-Informed Robot Learning [55.272458304179025]
  GS-Playground is a multi-modal simulation framework designed to accelerate end-to-end perceptual learning.<n>Our system achieves a breakthrough throughput of 104 FPS at 640x480 resolution, significantly lowering the barrier for large-scale visual RL.
  arXiv  Detail & Related papers  (2026-04-28T10:05:39Z)
• FLASH: Fast Learning via GPU-Accelerated Simulation for High-Fidelity Deformable Manipulation in Minutes [15.151803709340404]
  FLASH is a GPU-native simulation framework for contact-rich deformable manipulation.<n>It is built on an accurate NCP-based solver that enforces strict contact and deformation constraints.
  arXiv  Detail & Related papers  (2026-04-19T16:07:43Z)
• HorizonForge: Driving Scene Editing with Any Trajectories and Any Vehicles [63.88996084630768]
  Controllable driving scene generation is critical for realistic and scalable autonomous driving simulation.<n>We introduce HorizonForge, a unified framework that reconstructs scenes as editable Gaussian Splats and Meshes.<n>Experiments show that Gaussian-Mesh representation delivers substantially higher fidelity than alternative 3D representations.
  arXiv  Detail & Related papers  (2026-02-24T20:03:47Z)
• GeoPT: Scaling Physics Simulation via Lifted Geometric Pre-Training [86.70824679370524]
  We present GeoPT, a unified pre-trained model for general physics simulation based on lifted geometric pre-training.<n>The core idea is to augment geometry with synthetic dynamics, enabling dynamics-aware self-supervision without physics labels.
  arXiv  Detail & Related papers  (2026-02-23T22:32:08Z)
• GaussGym: An open-source real-to-sim framework for learning locomotion from pixels [78.05453137978132]
  We present a novel approach for photorealistic robot simulation that integrates 3D Gaussian Splatting as a drop-in within vectorized physics simulators.<n>This enables unprecedented speed -- exceeding 100,000 steps per second on consumer GPU.<n>We additionally demonstrate its applicability in a sim-to-real robotics setting.
  arXiv  Detail & Related papers  (2025-10-17T06:34:52Z)
• ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI [26.070510680936486]
  ManiSkill3 is the fastest state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation.<n>ManiSkill3 supports GPU parallelization of many aspects including simulation+rendering, heterogeneous simulation, pointclouds/voxels visual input, and more.
  arXiv  Detail & Related papers  (2024-10-01T06:10:39Z)
• Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting [32.59889755381453]
  Recent methods extend NeRF by incorporating tracked vehicle poses to animate vehicles, enabling photo-realistic view of dynamic urban street scenes. We introduce Street Gaussians, a new explicit scene representation that tackles these limitations. The proposed method consistently outperforms state-of-the-art methods across all datasets.
  arXiv  Detail & Related papers  (2024-01-02T18:59:55Z)
• Language Conditioned Traffic Generation [37.71751991840586]
  LCTGen is a large language model with a transformer-based decoder architecture that selects likely map locations from a dataset of maps. It produces an initial traffic distribution, as well as the dynamics of each vehicle. LCTGen outperforms prior work in both unconditional and conditional traffic scene generation in terms of realism and fidelity.
  arXiv  Detail & Related papers  (2023-07-16T05:10:32Z)
• DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
  We introduce a novel high-quality neural simulator referred to as DriveGAN. DriveGAN achieves controllability by disentangling different components without supervision. We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
  arXiv  Detail & Related papers  (2021-04-30T15:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.