VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model

• URL: http://arxiv.org/abs/2603.12655v1
• Date: Fri, 13 Mar 2026 04:56:43 GMT
• Title: VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model
• Abstract summary: VGGT-World is a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model features.<n>We show that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters.
• Score: 59.789011777899965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: World models that forecast scene evolution by generating future video frames devote the bulk of their capacity to photometric details, yet the resulting predictions often remain geometrically inconsistent. We present VGGT-World, a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model (GFM) features. Concretely, we repurpose the latent tokens of a frozen VGGT as the world state and train a lightweight temporal flow transformer to autoregressively predict their future trajectory. Two technical challenges arise in this high-dimensional (d=1024) feature space: (i) standard velocity-prediction flow matching collapses, and (ii) autoregressive rollout suffers from compounding exposure bias. We address the first with a clean-target (z-prediction) parameterization that yields a substantially higher signal-to-noise ratio, and the second with a two-stage latent flow-forcing curriculum that progressively conditions the model on its own partially denoised rollouts. Experiments on KITTI, Cityscapes, and TartanAir demonstrate that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters, establishing frozen GFM features as an effective and efficient predictive state for 3D world modeling.

Related papers

• Semigroup-JEPA: Latent Dynamics Consistency for Zero-Shot Physics Generalization [32.65135632960651]
  Joint-Embedding Predictive Architecture (JEPA) world models learn a compact latent representation of the world that supports prediction and planning.<n>We introduce SemiGroup-JEPA, which extends the LeWorldModel framework by supplying the parameter governing the physics to the temporal model.<n>We design dynamical tasks under different gravitational fields that, despite obeying the same physical law, exhibit qualitatively different dynamics.
  arXiv  Detail & Related papers  (2026-09-09T17:08:13Z)
• Physics-Guided Dual Decoding and Spectral Supervision for Global 3D Hydrometeor Prediction [2.1603484118769476]
  Three-dimensional hydrometeor forecasting is challenging due to the zero-inflated, long-tailed distributions of these variables.<n>We propose PredHydro-Net, a physics-guided dual-decoding framework that mitigates this smoothing.<n>By integrating wavelet-based frequency decoupling, spectral amplitude matching, and adversarial training, the model achieves a favorable trade-off between quantitative accuracy and spatial fidelity.
  arXiv  Detail & Related papers  (2026-06-07T10:38:37Z)
• Drift-Resistant Navigation World Model with Anchored Epipolar Guidance [81.52175010905665]
  We propose Drift-Resistant Navigation World Model, a generative model that mitigates both perceptual drift and geometric drift in conventional rollout-based navigation world models.<n>We first predict sparse future anchors that serve as stable long-range targets, and then generate intermediate frames within each chunk conditioned on both past context and future anchors.<n>Experiments on four benchmarks demonstrate consistent improvements over strong baselines in long-horizon visual quality, geometric consistency, and multi-view coherence.
  arXiv  Detail & Related papers  (2026-05-23T22:47:33Z)
• LeapTS: Rethinking Time Series Forecasting as Adaptive Multi-Horizon Scheduling [74.94985663101906]
  We present LeapTS, a novel framework that reformulates time series forecasting as a dynamic scheduling process over the prediction horizon.<n>We show that LeapTS improves overall forecasting performance by at least 7.4% while achieving a 2.6$times$ to 5.3$times$ inference speedup over representative Transformer-based models.
  arXiv  Detail & Related papers  (2026-05-11T09:54:02Z)
• RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space [51.441415833480505]
  RAYNOVA is a multiview world model for driving scenarios that employs a dual-causal autoregressive framework.<n>It constructs an isotropic-temporal representation across views, frames, and scales based on relative Plcker-ray positional encoding.
  arXiv  Detail & Related papers  (2026-02-24T08:41:40Z)
• When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Models [54.08784776767683]
  We present Physical-Conditioned World Model Attack (PhysCond-WMA), the first white-box world model attack that perturbs physical-condition channels.<n>PhysCond-WMA induces semantic, logic, or decision-level distortion while preserving perceptual fidelity.
  arXiv  Detail & Related papers  (2026-02-21T07:22:37Z)
• SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model [27.54931639768958]
  This paper introduces a novel architecture for trajectory-conditioned forecasting of future 3D scene occupancy.<n>Inspired by attention-based transformer architectures in foundational vision and language models such as GPT and VGGT, we employ a sparse occupancy representation that bypasses the intermediate bird's eye view (BEV) projection and its explicit geometric priors.<n>By avoiding both the finite-capacity constraint of discrete tokenization and the structural limitations of BEV representations, our method achieves state-of-the-art performance on the nuScenes benchmark for 1-3 second occupancy forecasting.
  arXiv  Detail & Related papers  (2025-11-27T02:48:45Z)
• Improving Tropical Cyclone Forecasting With Video Diffusion Models [0.0]
  Tropical cyclone (TC) forecasting is crucial for disaster preparedness and mitigation.<n>We present a novel application of video diffusion models for TC forecasting that explicitly models temporal dependencies through additional temporal layers.<n>Our approach enables the model to generate multiple frames simultaneously, better capturing cyclone evolution patterns.
  arXiv  Detail & Related papers  (2025-01-27T12:42:20Z)
• An Efficient Occupancy World Model via Decoupled Dynamic Flow and Image-assisted Training [50.71892161377806]
  DFIT-OccWorld is an efficient 3D occupancy world model that leverages decoupled dynamic flow and image-assisted training strategy.<n>Our model forecasts future dynamic voxels by warping existing observations using voxel flow, whereas static voxels are easily obtained through pose transformation.
  arXiv  Detail & Related papers  (2024-12-18T12:10:33Z)
• AdaOcc: Adaptive Forward View Transformation and Flow Modeling for 3D Occupancy and Flow Prediction [56.72301849123049]
  We present our solution for the Vision-Centric 3D Occupancy and Flow Prediction track in the nuScenes Open-Occ dataset challenge at CVPR 2024. Our innovative approach involves a dual-stage framework that enhances 3D occupancy and flow predictions by incorporating adaptive forward view transformation and flow modeling. Our method combines regression with classification to address scale variations in different scenes, and leverages predicted flow to warp current voxel features to future frames, guided by future frame ground truth.
  arXiv  Detail & Related papers  (2024-07-01T16:32:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.