Can Video World Models Track Unobserved World States?

• URL: http://arxiv.org/abs/2608.30692v1
• Date: Mon, 31 Aug 2026 12:29:24 GMT
• Title: Can Video World Models Track Unobserved World States?
• Abstract summary: Video world models are increasingly used as simulators, yet visual fidelity alone does not show that a model maintains the hidden state of the world.<n>We examine this gap with an action-conditioned video Shell Game, a visual analog of $S_5$ state tracking.<n>We find two mechanisms that do extrapolate, and both carry a state across chunks and revise it in place.
• Score: 40.869875403179535
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video world models are increasingly used as simulators, yet visual fidelity alone does not show that a model maintains the hidden state of the world. We examine this gap with an action-conditioned video Shell Game, a visual analog of $S_5$ state tracking that decouples visual rendering from compositing the hidden state underneath. Bidirectional and autoregressive Transformers, Mamba, and linear attention restricted to nonnegative transition eigenvalues all fit the training horizon of 5 swaps and then fall toward chance on longer swap chains (extrapolation) while still rendering plausible video with additional denoising steps providing no benefit. The pixel-based diffusion target never supervises the unseen hidden state, so the generated frames cannot carry it and the state has to live inside the architecture rather than in the tokens. For a Transformer, that architectural state is only an append-only KV cache, so the model has to re-derive the hidden arrangement from the whole history at every chunk. We find two mechanisms that do extrapolate, and both carry a state across chunks and revise it in place. Linear attention succeeds once its transition eigenvalues may be negative, and TTT with a nonlinear fast weight succeeds by updating the feature map through which it reads its own state. We further examine harder cases in dynamic world exploration tasks, and discuss the broader implications for building stateful video world models.

Related papers

• Marionette: Predicting World States, Rendering Geometry, Painting Appearance [30.170848813370444]
  We instantiate this idea as Marionette, a world model for interactive games with articulated characters.<n>A two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state.<n>A control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls.
  arXiv  Detail & Related papers  (2026-08-14T17:48:16Z)
• MASS: Multiplayer World Models with Authoritative Shared State [65.85526248184442]
  MASS (Multiplayer world models with Authoritative Shared State) is inspired by multiplayer game architectures.<n>A learned Logic Engine advances a global, authoritative typed state from joint actions.<n>A learned Rendering Engine generates independent and consistent views for any requested camera on demand.
  arXiv  Detail & Related papers  (2026-08-06T16:47:24Z)
• StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation [53.07918503798731]
  StatePlay is a novel state-aware game world model that predicts visual content and game states to promote mechanics-consistent generation.<n>Our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.
  arXiv  Detail & Related papers  (2026-07-29T10:49:30Z)
• Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers [50.3000377325823]
  We present WorldWeaver, a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers.<n>We ground these registers with supervision signals spanning individual agent status, global state views including bird's-eye views, and scene text.<n>Experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality.
  arXiv  Detail & Related papers  (2026-07-23T17:59:58Z)
• WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG [38.24292688170687]
  We propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations.<n>We show challenges in modeling semantically rich actions and maintaining long-horizon state consistency.
  arXiv  Detail & Related papers  (2026-03-24T17:58:25Z)
• Olaf-World: Orienting Latent Actions for Video World Modeling [100.96069208914957]
  Scaling action-controllable world models is limited by the scarcity of action labels.<n>We present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video.
  arXiv  Detail & Related papers  (2026-02-10T18:58:41Z)
• MAD: Motion Appearance Decoupling for efficient Driving World Models [94.40548866741791]
  We propose an efficient adaptation framework that converts generalist video models into controllable driving world models.<n>Key idea is to decouple motion learning from appearance synthesis.<n>Scaling to LTX, our MAD-LTX model outperforms all open-source competitors.
  arXiv  Detail & Related papers  (2026-01-14T12:52:23Z)
• ViPro-2: Unsupervised State Estimation via Integrated Dynamics for Guiding Video Prediction [5.931665251123055]
  We show that ViPro can infer states from observations without providing a full ground truth state in the beginning.<n>We extend the original Orbits dataset with a 3D variant to close the gap to real world scenarios.
  arXiv  Detail & Related papers  (2025-08-08T14:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.