InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving
Abstract Overview
InfiniVerse is a unified pipeline for generating long-horizon dynamic urban scenes for autonomous driving from a single multi-view observation. The method first reconstructs an explicit 3D occupancy representation, extends that scene autoregressively along user-defined trajectories, and then translates the occupancy grids into realistic multi-view videos with a fine-tuned video diffusion model. A central design is a bidirectional sketch-and-refine loop in which coarse occupancy guides video synthesis and generated frames are reprojected back to refine the 3D occupancy, aiming to maintain 2D-3D alignment over long rollouts. Experiments on Waymo Open Dataset and nuScenes evaluate generation quality, temporal consistency, and occupancy reconstruction, with additional analyses of rollout length and component ablations.
Novelty
The paper's main novelty is the integration of explicit occupancy-based world modeling with video diffusion in a closed reciprocal loop, rather than treating geometry and video generation as separate stages. It also introduces a hierarchical sketch-guided scene extrapolation mechanism that enables controllable long-range generation from only a single multi-view input.
Results
On the reported benchmarks, InfiniVerse achieves state-of-the-art generation quality with FID 6.40 and FVD 67.97, and the paper reports that degradation over longer rollout horizons is slower than Vista. Ablation results show that adding the 3D branch, reciprocal loop, and region-aware loss progressively improves video quality, and the refined voxel reconstruction reaches 34.31% IoU and 20.12% mIoU.
Key Points
- InfiniVerse builds an explicit 3D occupancy world from a single multi-view frame and extends it along arbitrary trajectories for long-horizon scene generation.
- The method couples occupancy generation and video diffusion through a bidirectional sketch-and-refine loop to improve temporal stability and geometry-appearance alignment.
- Experiments on Waymo and nuScenes report strong benchmark performance and better long-rollout stability than the reproduced Vista baseline.