AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond

• URL: http://arxiv.org/abs/2605.26113v1
• Date: Mon, 25 May 2026 17:59:48 GMT
• Title: AnyScene: Towards Highly Controllable Driving Scene Generation at Anywhere and Beyond
• Abstract summary: AnyScene is a unified occupancy-centric framework for driving scene generation.<n>It generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer.<n>Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation.
• Score: 34.2135689224471
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating high-fidelity and controllable synthetic data is critical for advancing end-to-end autonomous driving, particularly for addressing the long tail of rare safety-critical scenarios. Existing occupancy-guided methods typically rely on shallow conditioning mechanisms and reference-frame-dependent video synthesis, which limits fine-grained controllability from arbitrary BEV layouts and restricts their applicability for scalable simulation. In this paper, we propose AnyScene, a unified occupancy-centric framework for driving scene generation. AnyScene generates semantic occupancy sequences from BEV layouts through a Spatial-Temporal Occupancy Diffusion Transformer that jointly tokenizes BEV and occupancy features in an autoregressive manner. This design enables precise controllability from cross-dataset and user-defined BEV inputs while naturally supporting long-horizon generation. Building upon the generated occupancy, a Geometry-Grounded View Expansion module treats occupancy as the canonical spatial representation and synthesizes temporally consistent multi-view driving videos in a reference-free and autoregressive fashion, supporting flexible camera configurations at inference time. Extensive experiments demonstrate that AnyScene achieves state-of-the-art performance in both occupancy and video generation. It exhibits strong generalization to unseen and customized layouts, and provides measurable benefits for downstream tasks such as sparse-view 3D reconstruction.

Related papers

• CARLA-GS: Decoupling Representation, Reasoning, and Physics Simulation for Autonomous Driving Corner-Case Synthesis [3.0621944624163695]
  Corner-case generation is inherently a multi-case problem spanning visual representation, scene reasoning, and vehicle trajectory generation and control.<n>To unify these aspects within a single framework, we propose CARLA-GS, a modular corner-case synthesis pipeline.<n>This design enables high-level semantic reasoning, low-level physically executable motion, and corner-case generation within a unified pipeline.
  arXiv  Detail & Related papers  (2026-07-08T16:20:19Z)
• DriveWeaver: Point-Conditioned Video Inpainting for Controllable Vehicle Insertion in Autonomous Driving Simulation [9.201369685231532]
  DriveWeaver is a novel framework for controllable vehicle insertion in autonomous driving simulation.<n>We perform video inpainting conditioned on vehicle point clouds to generate temporally consistent vehicles.<n>Our method outperforms existing baselines in visual realism and geometric consistency.
  arXiv  Detail & Related papers  (2026-06-30T16:23:32Z)
• InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving [70.7737035270887]
  We introduce InfiniVerse, a unified pipeline for synthesis of dynamic urban scenes from a single frame.<n>InfiniVerse achieves state-of-the-art performance with a FID of 6.4 and FVD of 67.97, significantly outperforming existing benchmarks in both duration and stability.
  arXiv  Detail & Related papers  (2026-06-30T04:08:22Z)
• DriveCtrl: Conditioned Sim-to-Real Driving Video Generation [16.424889754682727]
  DriveCtrl is a controllable sim-to-real video generation framework for realistic driving video synthesis.<n>We introduce a scalable data generation pipeline that transforms simulator videos into realistic driving footage matching the visual style of a target real-world dataset.
  arXiv  Detail & Related papers  (2026-05-14T17:29:35Z)
• FAR-Drive: Frame-AutoRegressive Video Generation in Closed-Loop Autonomous Driving [11.275815014211046]
  Building a learning-based closed-loop simulator for autonomous driving poses three major challenges.<n>Long-horizon temporal and cross-view consistency, mitigating autoregressive degradation under iterative self-conditioning, and satisfying low-latency inference constraints.<n>We propose FAR-Drive, a frame-level autoregressive video generation framework for autonomous driving.
  arXiv  Detail & Related papers  (2026-03-16T07:40:13Z)
• Optimization-Guided Diffusion for Interactive Scene Generation [52.23368750264419]
  We present OMEGA, an optimization-guided, training-free framework that enforces structural consistency and interaction awareness during diffusion-based sampling.<n>We show that OMEGA improves generation realism, consistency, and controllability, increasing the ratio of physically and behaviorally valid scenes.<n>Our approach can also generate $5times$ more near-collision frames with a time-to-collision under three seconds.
  arXiv  Detail & Related papers  (2025-12-08T15:56:18Z)
• Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method [54.461213497603154]
  Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities.<n>Nuplan-Occ is the largest occupancy dataset to date, constructed from the widely used Nuplan benchmark.<n>We develop a unified framework that jointly synthesizes high-quality occupancy, multi-view videos, and LiDAR point clouds.
  arXiv  Detail & Related papers  (2025-10-27T03:52:45Z)
• X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability [49.4647778989539]
  X-Scene is a novel framework for large-scale driving scene generation.<n>It achieves both geometric intricacy and appearance fidelity, while offering flexible controllability.<n>X-Scene significantly advances controllability and fidelity for large-scale driving scene generation.
  arXiv  Detail & Related papers  (2025-06-16T14:43:18Z)
• UniScene: Unified Occupancy-centric Driving Scene Generation [73.22859345600192]
  We introduce UniScene, the first unified framework for generating three key data forms - semantic occupancy, video, and LiDAR.<n>UniScene employs a progressive generation process that decomposes the complex task of scene generation into two hierarchical steps.<n>Extensive experiments demonstrate that UniScene outperforms previous SOTAs in the occupancy, video, and LiDAR generation.
  arXiv  Detail & Related papers  (2024-12-06T21:41:52Z)
• Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention [61.3281618482513]
  We present CogDriving, a novel network designed for synthesizing high-quality multi-view driving videos.<n>CogDriving leverages a Diffusion Transformer architecture with holistic-4D attention modules, enabling simultaneous associations across the dimensions.<n>CogDriving demonstrates strong performance on the nuScenes validation set, achieving an FVD score of 37.8, highlighting its ability to generate realistic driving videos.
  arXiv  Detail & Related papers  (2024-12-04T18:02:49Z)
• DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation [10.296670127024045]
  DriveScape is an end-to-end framework for multi-view, 3D condition-guided video generation. Our Bi-Directional Modulated Transformer (BiMot) ensures precise alignment of 3D structural information. DriveScape excels in video generation performance, achieving state-of-the-art results on the nuScenes dataset with an FID score of 8.34 and an FVD score of 76.39.
  arXiv  Detail & Related papers  (2024-09-09T09:43:17Z)
• BEVWorld: A Multimodal World Simulator for Autonomous Driving via Scene-Level BEV Latents [56.33989853438012]
  We propose BEVWorld, a framework that transforms multimodal sensor inputs into a unified and compact Bird's Eye View latent space for holistic environment modeling.<n>The proposed world model consists of two main components: a multi-modal tokenizer and a latent BEV sequence diffusion model.
  arXiv  Detail & Related papers  (2024-07-08T07:26:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.