Orbis 2: A Hierarchical World Model for Driving

• URL: http://arxiv.org/abs/2607.15898v1
• Date: Fri, 17 Jul 2026 12:12:33 GMT
• Title: Orbis 2: A Hierarchical World Model for Driving
• Abstract summary: We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales.<n>This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations.<n>We introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing.
• Score: 21.534159792492186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks. We present a hierarchical driving world model that factorizes future prediction across two levels operating at distinct temporal and abstraction scales: a high-level predictor that forecasts coarse scene structure over extended temporal horizons, and a low-level generator that produces detailed predictions conditioned on the high-level output. This decomposition yields high perceptual fidelity while also capturing strong spatial and semantic representations. We further show that pretraining with a diffusion forcing objective yields substantially richer internal representations than the standard teacher forcing objective, while teacher forcing -- predicting only the next frame from clean context -- produces more stable autoregressive rollouts. We therefore introduce a generic two-stage training paradigm that pretrains the model with diffusion forcing and fine-tunes with teacher forcing, combining the representational benefits of the former with the rollout stability of the latter. Our approach achieves state-of-the-art results across the standard suite of driving world model evaluations on established benchmarks, including long-horizon generation fidelity, steering responsiveness evaluated on counterfactual scenarios, and internal representation quality. Project page with code, demo, checkpoints and qualitative results: https://lmb-freiburg.github.io/orbis2.github.io/

Related papers

• HyWorldVLA: A Vision-Language-Action Model with Hybrid World Modeling for Autonomous Driving [11.703169226106288]
  We propose HyWorldVLA, a hybrid world-VLA framework that unifies pixel-level supervision and latent representation learning.<n>In the pre-training stage, HyWorldVLA predicts latent video encoded by a pre-trained video VAE, while simultaneously reconstructing video frames to provide precise pixel-level grounding.<n>During the subsequent co-fine-tuning phase, the model exclusively predicts latent features, which are fed into an action expert to generate trajectories.
  arXiv  Detail & Related papers  (2026-07-23T07:11:41Z)
• Flow Matching in Feature Space for Stochastic World Modeling [21.159664319255665]
  We propose FlowWM, a world model that performs flow matching directly within pretrained feature space.<n>This is challenging because pretrained features are substantially high-dimensional, making standard diffusion recipes suboptimal.<n>We evaluate FlowWM on two benchmarks: a synthetic benchmark for systematic evaluation of accuracy and diversity, and a real-world benchmark FuturePerception.
  arXiv  Detail & Related papers  (2026-06-27T19:35:27Z)
• ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model [53.15040805435013]
  Vision-language models (VLMs) provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames.<n>We propose a VLM-guided JEPA-style latent world modeling framework that combines dense-frame dynamics modeling with long-horizon semantic guidance.
  arXiv  Detail & Related papers  (2026-03-23T17:59:42Z)
• Open-World Motion Forecasting [15.508091258980143]
  Motion forecasting aims to predict the future trajectories of dynamic agents in the scene, enabling autonomous vehicles to reason about scene evolution.<n>Existing approaches operate under the closed-world regime and assume fixed object taxonomy as well as access to high-quality perception.<n>We introduce open-world motion forecasting, a novel setting in which new object classes are sequentially introduced over time and future object trajectories are estimated directly from camera images.
  arXiv  Detail & Related papers  (2026-03-10T09:35:08Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignment [45.21358158991569]
  VLA models can predict environmental dynamics, known as world modeling.<n>Current approaches face two main issues: 1) The training objective forces models to over-emphasize pixel-level reconstruction, which constrains semantic learning and generalization, and 2) Reliance on predicted future observations during inference often leads to error accumulation.<n>We introduce Future Representation Alignment via Parallel Progressive Expansion (FRAPPE) to address these challenges.
  arXiv  Detail & Related papers  (2026-02-19T11:00:46Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method [54.461213497603154]
  Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities.<n>Nuplan-Occ is the largest occupancy dataset to date, constructed from the widely used Nuplan benchmark.<n>We develop a unified framework that jointly synthesizes high-quality occupancy, multi-view videos, and LiDAR point clouds.
  arXiv  Detail & Related papers  (2025-10-27T03:52:45Z)
• Generalist Forecasting with Frozen Video Models via Latent Diffusion [35.96406989431198]
  We show a strong correlation between a vision model's perceptual ability and its generalist forecasting performance over short time horizons.<n>Our results highlight the value of bridging representation learning and generative modeling for temporally grounded video understanding.
  arXiv  Detail & Related papers  (2025-07-18T14:14:19Z)
• OPUS: Occupancy Prediction Using a Sparse Set [64.60854562502523]
  We present a framework to simultaneously predict occupied locations and classes using a set of learnable queries. OPUS incorporates a suite of non-trivial strategies to enhance model performance. Our lightest model achieves superior RayIoU on the Occ3D-nuScenes dataset at near 2x FPS, while our heaviest model surpasses previous best results by 6.1 RayIoU.
  arXiv  Detail & Related papers  (2024-09-14T07:44:22Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.