Towards Spatial Supersensing in the Wild

• URL: http://arxiv.org/abs/2607.13681v1
• Date: Wed, 15 Jul 2026 10:24:51 GMT
• Title: Towards Spatial Supersensing in the Wild
• Abstract summary: We introduce $textbfVSI-Super-Wild, a benchmark for evaluating spatial supersensing over long temporal horizons.<n>Results show that models consistently fail at tasks that require coherent world-state tracking over time.<n>We characterize how performance degrades with world-state and temporal horizon, and diagnose four failure modes: spatial collapse, semantic shortcuts, complexity update, and instance confusion.
• Score: 35.413360698005285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans can efficiently parse continuous sensory streams, from hours to years, scaffolding an internal world model that grounds spatial reasoning and prediction. To mimic this capacity, spatial supersensing challenges multimodal models to move beyond linguistic understanding toward true world modeling. However, their benchmark relies on synthetic long videos, formed by concatenating random short clips, and is mostly limited to household scenes, leaving real-world continuity and diversity underexplored. To address the gap, we introduce $\textbf{VSI-Super-Wild}$, a large-scale benchmark for evaluating spatial supersensing over long temporal horizons in diverse in-the-wild scenes. Notably, inspired by cognitive studies on how humans structure experience, we systematically probe the full triad of world state: the agent (observer), objects (scene items), and the environment (places and global layout). In total, VSI-Super-Wild contains $\textbf{6,980}$ human-verified question-answer pairs derived from $\textbf{442}$ real-world videos spanning 8 scene categories, including long-form recordings exceeding 4 hours. Results on VSI-Super-Wild expose a fundamental disconnect: despite advances in static image understanding, models consistently fail at tasks that require coherent world-state tracking over time. We characterize how performance degrades with world-state complexity and temporal horizon, and diagnose four failure modes: spatial collapse, semantic shortcuts, insufficient update, and instance confusion. This taxonomy reveals that models lack mechanisms to bind objects, agents, and environments into a unified spatial world model, a fundamental gap that defines the path forward for spatial supersensing.

Related papers

• World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning [18.658825757342544]
  Vision-language models (VLMs) have shown strong performance on static visual understanding.<n>They still struggle with dynamic spatial reasoning that requires imagining how scenes evolve under egocentric motion.<n>We propose World2VLM, a training framework that distills spatial imagination from a generative world model into a vision-language model.
  arXiv  Detail & Related papers  (2026-04-29T17:48:01Z)
• INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modeling [44.09983529522167]
  INSPATIO-WORLD is a novel real-time framework capable of recovering and generating high-fidelity interactive scenes from a single video.<n>Implicit Spatiotemporal Cache aggregates reference and historical observations into a latent world representation.<n>Explicit Spatial Constraint Module enforces geometric structure and translates user interactions into precise and physically plausible camera trajectories.
  arXiv  Detail & Related papers  (2026-04-08T15:31:22Z)
• Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World [49.80040477190479]
  Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time.<n>We introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets.<n>We find that existing models cannot simultaneously maintain strong performance in both-temporal reasoning and dynamic object grounding.
  arXiv  Detail & Related papers  (2026-03-13T07:42:16Z)
• Beyond Pixel Histories: World Models with Persistent 3D State [50.4601060508243]
  PERSIST is a new paradigm of world model which simulates the evolution of a latent 3D scene.<n>We show substantial improvements in spatial memory, 3D consistency, and long-horizon stability over existing methods.
  arXiv  Detail & Related papers  (2026-03-03T19:58:31Z)
• TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model [53.555353366322464]
  We present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system.<n>Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible synthesis systems.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• Cambrian-S: Towards Spatial Supersensing in Video [78.46305169769884]
  We frame spatial supersensing as four stages beyond linguistic-only understanding: semantic perception, streaming event cognition, implicit 3D spatial cognition, and predictive world modeling.<n>To drive progress in spatial supersensing, we present-SUPER, a two-part benchmark: VSR (long-horizon visual spatial recall) and VSC (continual visual spatial counting)<n>We then test data scaling limits by curating-590K and training Cambrian-S, achieving +30% absolute improvement on arbitrarily general capabilities.<n>We propose predictive sensing as a path forward, presenting a proof-of-concept in which a self-supervised
  arXiv  Detail & Related papers  (2025-11-06T18:55:17Z)
• OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling [86.12242953301121]
  We introduce OmniWorld, a large-scale, multi-domain, multi-modal dataset specifically designed for 4D world modeling.<n>Compared with existing synthetic datasets, OmniWorld-Game provides richer modality coverage, larger scale, and more realistic dynamic interactions.<n>We establish a challenging benchmark that exposes the limitations of current state-of-the-art (SOTA) approaches in modeling complex 4D environments.
  arXiv  Detail & Related papers  (2025-09-15T17:59:19Z)
• Hieros: Hierarchical Imagination on Structured State Space Sequence World Models [4.922995343278039]
  Hieros is a hierarchical policy that learns time abstracted world representations and imagines trajectories at multiple time scales in latent space. We show that our approach outperforms the state of the art in terms of mean and median normalized human score on the Atari 100k benchmark.
  arXiv  Detail & Related papers  (2023-10-08T13:52:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.