Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks

• URL: http://arxiv.org/abs/2605.18194v1
• Date: Mon, 18 May 2026 10:32:56 GMT
• Title: Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks
• Abstract summary: We introduce an Anchor-Based Embodied Spatial Decomposition Chain-of-Thought (CoT)<n>CoT guides the MLLM through a "geometric-to-semantic" projection, forcing it to first establish B's local coordinate system and then dynamically weight visual and auditory modalities based on whether A falls within B's visual frustum.
• Score: 16.03812136589077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Multi-Modal Large Language Models (MLLMs) demonstrate impressive capabilities in general reasoning, their embodied spatial intelligence remains hampered by a "Cartesian Illusion" - a reliance on text-based probability distributions that lack grounded, 3D topological understanding. This limitation is starkly exposed in multi-agent environments, which demand more than just scene perception; they require second-order Theory of Mind (ToM). Specifically, an Agent A must be able to infer Agent B's belief about the environment, governed strictly by Agent B's physical orientation and sensory limitations. In this paper, we probe the limits of two-stage spatial inference in MLLMs through a novel audio-visual task: requiring Agent A to predict Agent B's estimation of A's relative location. To solve this, we propose an Epistemic Sensory Bottleneck module that abandons rigid, rule-based coordinate transformations. Instead, we introduce an Anchor-Based Embodied Spatial Decomposition Chain-of-Thought (CoT). This guides the MLLM through a "geometric-to-semantic" projection, forcing it to first establish B's local coordinate system and then dynamically weight visual and auditory modalities based on whether A falls within B's visual frustum. Extensive evaluations reveal that while current MLLMs fundamentally struggle with spatial symmetry and out-of-view ambiguities (establishing a rigorous zero-shot baseline of 42% accuracy), our sensory-bounded reasoning chain robustly outperforms pure egocentric and allocentric baselines. By systematically benchmarking these perceptual bottlenecks, our work exposes the current limits of MLLM spatial reasoning and establishes a foundational paradigm for epistemic, modality-aware inference in Embodied AI.

Related papers

• S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence [77.5121349100339]
  textscS-Agent reshapes spatial perception into scene-centric understanding beyond frame-centric recognition.<n>textscS-Agent casts the VLM as a semantic planner that decides what evidence is needed.<n>textscS-Agent consistently improves both open-source and closed-source VLMs in a training-free manner.
  arXiv  Detail & Related papers  (2026-06-18T17:34:55Z)
• VertiCue-Bench: Diagnosing Whether MLLMs Use Height Cues to Resolve 2D Ambiguity in Remote Sensing Natural Scenes [17.920935934613368]
  VertiCue-Bench is the first diagnostic benchmark for CHM-grounded geospatial reasoning.<n>We evaluate 14 state-of-the-art general and remote-sensing-specialized MLLMs.<n>Overall, VertiCue-Bench exposes a critical geometry-to-semantics gap in natural scene understanding.
  arXiv  Detail & Related papers  (2026-05-25T12:30:33Z)
• ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop [55.468404995694975]
  We develop a benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson.<n>We conduct experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts.<n>Unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality.
  arXiv  Detail & Related papers  (2026-05-18T17:59:02Z)
• SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments [19.997461654311994]
  We introduce SpaMEM, a large-scale diagnostic benchmark that isolates the mechanics of spatial belief evolution.<n>SpaMEM is built on a physically grounded dataset with 10,601,392 high-fidelity images across four modalities.<n>We formalize embodied spatial reasoning as a three-level hierarchy with 15 diagnostic tasks.
  arXiv  Detail & Related papers  (2026-04-24T10:06:41Z)
• SpatiO: Adaptive Test-Time Orchestration of Vision-Language Agents for Spatial Reasoning [18.3204772691015]
  spatial reasoning requires flexibly coordinating different strategies depending on the input.<n>Most existing approaches rely on a single reasoning pipeline that implicitly learns a fixed spatial prior.<n>We introduce SpatiO, a heterogeneous multi-agent framework for spatial reasoning that coordinates multiple vision-language specialists with complementary inductive biases.
  arXiv  Detail & Related papers  (2026-04-23T01:19:37Z)
• SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning [67.67774742200626]
  spatial intelligence, which refers to the ability to reason about geometric and physical structure from visual observations, is a core challenge for large language models.<n>We propose a unified multimodal generation framework that integrates textual reasoning with visual imagination.<n>Our framework adopts a divide-and-conquer strategy, using text chain-of-thought for high-level semantic planning and the visual imagination for geometry-sensitive state transformation and consistency preservation.
  arXiv  Detail & Related papers  (2026-04-19T11:21:59Z)
• Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models [62.932580559941414]
  Vision-Language Models (VLMs) frequently "hallucinate" - generate plausible yet factually incorrect statements.<n>We propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model's computational cognition.
  arXiv  Detail & Related papers  (2026-03-16T17:20:38Z)
• Cognitively-Inspired Tokens Overcome Egocentric Bias in Multimodal Models [0.0]
  Multimodal language models (MLMs) fail at spatial reasoning that requires adopting another agent's visual perspective.<n>Inspired by human spatial cognition, we introduce perspective tokens that encode orientation through either (1) embodied body-keypoint cues or (2) abstract representations supporting mental rotation.<n>Across synthetic and naturalistic benchmarks, perspective tokens improve accuracy, with rotation-based tokens generalizing to non-human reference agents.
  arXiv  Detail & Related papers  (2026-01-23T00:21:27Z)
• SpatialTree: How Spatial Abilities Branch Out in MLLMs [109.32057088014942]
  We introduce a cognitive-science-inspired hierarchy that organizes spatial abilities into four levels: low-level perception (L1), mental mapping (L2), simulation (L3), and agentic competence (L4)<n>We find that naive RL that encourages extensive "thinking" is unreliable: it helps complex reasoning but hurts intuitive perception.<n>We propose a simple auto-think strategy that suppresses unnecessary deliberation, enabling RL to consistently improve performance across all levels.
  arXiv  Detail & Related papers  (2025-12-23T18:59:46Z)
• SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagery [64.67498968405327]
  SpatialDreamer is a reinforcement learning framework that enables spatial reasoning through a closedloop process of active exploration.<n>GeoPO introduces tree-structured sampling and step-level reward estimation with consistency geometric constraints.
  arXiv  Detail & Related papers  (2025-12-08T17:20:50Z)
• Geometrically-Constrained Agent for Spatial Reasoning [53.93718394870856]
  Vision Language Models exhibit a fundamental semantic-to-geometric gap in spatial reasoning.<n>Current paradigms fail to bridge this gap.<n>We propose a training-free agentic paradigm that resolves this gap by introducing a formal task constraint.
  arXiv  Detail & Related papers  (2025-11-27T17:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.