Delta-Triplane Transformers as Occupancy World Models

• URL: http://arxiv.org/abs/2503.07338v3
• Date: Sat, 27 Sep 2025 18:26:34 GMT
• Title: Delta-Triplane Transformers as Occupancy World Models
• Authors: Haoran Xu, Peixi Peng, Guang Tan, Yiqian Chang, Yisen Zhao, Yonghong Tian,
• Abstract summary: Occupancy World Models (OWMs) aim to predict future scenes via 3D voxelized representations of the environment to support intelligent motion planning.<n>We propose Delta-Triplane Transformers (DTT), a novel 4D OWM for autonomous driving, that introduces two key innovations.<n>DTT delivers a 1.44$times$ speedup (26 FPS) over the state of the art, improves mean IoU to 30.85, and reduces the mean absolute planning error to 1.0 meters.
• Score: 57.16979927973973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Occupancy World Models (OWMs) aim to predict future scenes via 3D voxelized representations of the environment to support intelligent motion planning. Existing approaches typically generate full future occupancy states from VAE-style latent encodings, which can be computationally expensive and redundant. We propose Delta-Triplane Transformers (DTT), a novel 4D OWM for autonomous driving, that introduces two key innovations: (1) a triplane based representation that encodes 3D occupancy more compactly than previous approaches, and (2) an incremental prediction strategy for OWM that models {\em changes} in occupancy rather than dealing with full states. The core insight is that changes in the compact 3D latent space are naturally sparser and easier to model, enabling higher accuracy with a lighter-weight architecture. Building on this representation, DTT extracts multi-scale motion features from historical data and iteratively predict future triplane deltas. These deltas are combined with past states to decode future occupancy and ego-motion trajectories. Extensive experiments demonstrate that DTT delivers a 1.44$\times$ speedup (26 FPS) over the state of the art, improves mean IoU to 30.85, and reduces the mean absolute planning error to 1.0 meters. Demo videos are provided in the supplementary material.

Related papers

• Towards Efficient and Effective Multi-Camera Encoding for End-to-End Driving [54.85072592658933]
  We present Flex, an efficient and effective scene encoder that addresses the computational bottleneck of processing high-volume multi-camera data in autonomous driving.<n>By design, our approach is geometry-agnostic, learning a compact scene representation directly from data without relying on the explicit 3D inductive biases.<n>Our findings challenge the prevailing assumption that 3D priors are necessary, demonstrating that a data-driven, joint encoding strategy offers a more scalable, efficient and effective path for future autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-11T18:59:46Z)
• World4Drive: End-to-End Autonomous Driving via Intention-aware Physical Latent World Model [18.56171397212777]
  We present World4Drive, an end-to-end autonomous driving framework that employs vision foundation models to build latent world models.<n>World4Drive achieves state-of-the-art performance without manual perception annotations on open-loop nuScenes and closed-loop NavSim benchmarks.
  arXiv  Detail & Related papers  (2025-07-01T09:36:38Z)
• LMPOcc: 3D Semantic Occupancy Prediction Utilizing Long-Term Memory Prior from Historical Traversals [4.970345700893879]
  Longterm Memory Prior Occupancy (LMPOcc) is the first 3D occupancy prediction methodology that exploits long-term memory priors derived from historical perceptual outputs. We introduce a plug-and-play architecture that integrates long-term memory priors to enhance local perception while simultaneously constructing global occupancy representations.
  arXiv  Detail & Related papers  (2025-04-18T09:58:48Z)
• Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction [26.204219108066454]
  We propose novel diffusion models (MMTwin) for multimodal 3D hand trajectory prediction.<n>MMTwin is designed to absorb multimodal information as input encompassing 2D RGB images, 3D point clouds, past hand waypoints, and text prompt.<n>Two latent diffusion models, the egomotion diffusion and the HTP diffusion as twins, are integrated into MMTwin to predict camera egomotion and future hand trajectories concurrently.
  arXiv  Detail & Related papers  (2025-04-10T01:29:50Z)
• EMoTive: Event-guided Trajectory Modeling for 3D Motion Estimation [59.33052312107478]
  Event cameras offer possibilities for 3D motion estimation through continuous adaptive pixel-level responses to scene changes. This paper presents EMove, a novel event-based framework that models-uniform trajectories via event-guided parametric curves. For motion representation, we introduce a density-aware adaptation mechanism to fuse spatial and temporal features under event guidance. The final 3D motion estimation is achieved through multi-temporal sampling of parametric trajectories, flows and depth motion fields.
  arXiv  Detail & Related papers  (2025-03-14T13:15:54Z)
• Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving [22.832008530490167]
  We propose a semi-supervised vision-centric 3D occupancy world model, PreWorld, to leverage the potential of 2D labels.<n>PreWorld achieves competitive performance across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks.
  arXiv  Detail & Related papers  (2025-02-11T07:12:26Z)
• Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles [81.29018359825872]
  This paper consolidates a set of good practices to finetune large pretrained models for a real-world task.<n>Specifically, we develop several strategies to account for discrepancies between the synthetic data and real driving data.<n>Our insights lead to effective finetuning that results in a $68.8%$ reduction in FID for novel view synthesis over prior arts.
  arXiv  Detail & Related papers  (2024-12-19T03:39:13Z)
• An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training [50.71892161377806]
  DFIT-OccWorld is an efficient 3D occupancy world model that leverages decoupled dynamic flow and image-assisted training strategy.<n>Our model forecasts future dynamic voxels by warping existing observations using voxel flow, whereas static voxels are easily obtained through pose transformation.
  arXiv  Detail & Related papers  (2024-12-18T12:10:33Z)
• GaussianWorld: Gaussian World Model for Streaming 3D Occupancy Prediction [67.81475355852997]
  3D occupancy prediction is important for autonomous driving due to its comprehensive perception of the surroundings.<n>We propose a world-model-based framework to exploit the scene evolution for perception.<n>Our framework improves the performance of the single-frame counterpart by over 2% in mIoU without introducing additional computations.
  arXiv  Detail & Related papers  (2024-12-13T18:59:54Z)
• DOME: Taming Diffusion Model into High-Fidelity Controllable Occupancy World Model [14.996395953240699]
  DOME is a diffusion-based world model that predicts future occupancy frames based on past occupancy observations. The ability of this world model to capture the evolution of the environment is crucial for planning in autonomous driving.
  arXiv  Detail & Related papers  (2024-10-14T12:24:32Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)
• Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving [15.100104512786107]
  Drive-OccWorld adapts a visioncentric- 4D forecasting world model to end-to-end planning for autonomous driving.<n>We propose injecting flexible action conditions, such as velocity, steering angle, trajectory, and commands, into the world model to enable controllable generation.<n>Our method can generate plausible and controllable 4D occupancy, paving the way for advancements in driving world generation and end-to-end planning.
  arXiv  Detail & Related papers  (2024-08-26T11:53:09Z)
• OccSora: 4D Occupancy Generation Models as World Simulators for Autonomous Driving [62.54220021308464]
  We propose a diffusion-based 4D occupancy generation model, OccSora, to simulate the development of the 3D world for autonomous driving. OccSora can generate 16s-videos with authentic 3D layout and temporal consistency, demonstrating its ability to understand the spatial and temporal distributions of driving scenes.
  arXiv  Detail & Related papers  (2024-05-30T17:59:42Z)
• DriveWorld: 4D Pre-trained Scene Understanding via World Models for Autonomous Driving [67.46481099962088]
  Current vision-centric pre-training typically relies on either 2D or 3D pre-text tasks, overlooking the temporal characteristics of autonomous driving as a 4D scene understanding task. We introduce emphcentricDriveWorld, which is capable of pre-training from multi-camera driving videos in atemporal fashion. DriveWorld delivers promising results on various autonomous driving tasks.
  arXiv  Detail & Related papers  (2024-05-07T15:14:20Z)
• Improving Trajectory Prediction in Dynamic Multi-Agent Environment by Dropping Waypoints [9.385936248154987]
  Motion prediction systems must learn spatial and temporal information from the past to forecast the future trajectories of the agent. We propose Temporal Waypoint Dropping (TWD) that explicitly incorporates temporal dependencies during the training of a trajectory prediction model. We evaluate our proposed approach on three datasets: NBA Sports VU, ETH-UCY, and TrajNet++.
  arXiv  Detail & Related papers  (2023-09-29T15:48:35Z)
• Uncertainty-aware State Space Transformer for Egocentric 3D Hand Trajectory Forecasting [79.34357055254239]
  Hand trajectory forecasting is crucial for enabling a prompt understanding of human intentions when interacting with AR/VR systems. Existing methods handle this problem in a 2D image space which is inadequate for 3D real-world applications. We set up an egocentric 3D hand trajectory forecasting task that aims to predict hand trajectories in a 3D space from early observed RGB videos in a first-person view.
  arXiv  Detail & Related papers  (2023-07-17T04:55:02Z)
• VAD: Vectorized Scene Representation for Efficient Autonomous Driving [44.070636456960045]
  VAD is an end-to-end vectorized paradigm for autonomous driving. VAD exploits the vectorized agent motion and map elements as explicit instance-level planning constraints. VAD runs much faster than previous end-to-end planning methods.
  arXiv  Detail & Related papers  (2023-03-21T17:59:22Z)
• Pedestrian 3D Bounding Box Prediction [83.7135926821794]
  We focus on 3D bounding boxes, which are reasonable estimates of humans without modeling complex motion details for autonomous vehicles. We suggest this new problem and present a simple yet effective model for pedestrians' 3D bounding box prediction. This method follows an encoder-decoder architecture based on recurrent neural networks.
  arXiv  Detail & Related papers  (2022-06-28T17:59:45Z)
• Graph-based Spatial Transformer with Memory Replay for Multi-future Pedestrian Trajectory Prediction [13.466380808630188]
  We propose a model to forecast multiple paths based on a historical trajectory. Our method can exploit the spatial information as well as correct the temporally inconsistent trajectories. Our experiments show that the proposed model achieves state-of-the-art performance on multi-future prediction and competitive results for single-future prediction.
  arXiv  Detail & Related papers  (2022-06-12T10:25:12Z)
• A Spatio-temporal Transformer for 3D Human Motion Prediction [39.31212055504893]
  We propose a Transformer-based architecture for the task of generative modelling of 3D human motion. We empirically show that this effectively learns the underlying motion dynamics and reduces error accumulation over time observed in auto-gressive models.
  arXiv  Detail & Related papers  (2020-04-18T19:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.