ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop

• URL: http://arxiv.org/abs/2605.18746v2
• Date: Mon, 25 May 2026 08:34:52 GMT
• Title: ESI-Bench: Towards Embodied Spatial Intelligence that Closes the Perception-Action Loop
• Abstract summary: We develop a benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson.<n>We conduct experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts.<n>Unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality.
• Score: 55.468404995694975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial intelligence unfolds through a perception-action loop: agents act to acquire observations, and reason about how observations vary as a function of action. Rather than passively processing what is seen, they actively uncover what is unseen - occluded structure, dynamics, containment, and functionality that cannot be resolved from passive sensing alone. We move beyond prior formulations of spatial intelligence that assume oracle observations by recasting the observer as an actor. We introduce ESI-BENCH, a comprehensive benchmark for embodied spatial intelligence spanning 10 task categories and 29 subcategories built on OmniGibson, grounded in Spelke's core knowledge systems. Agents must decide what abilities to deploy - perception, locomotion, and manipulation - and how to sequence them to actively accumulate task-relevant evidence. We conduct extensive experiments on state-of-the-art MLLMs and find that active exploration substantially outperforms passive counterparts, with agents spontaneously discovering emergent spatial strategies without explicit instructions, while random multi-view often adds noise rather than signal despite consuming far more images. Most failures stem not from weak perception but from action blindness: poor action choices lead to poor observations, which in turn drive cascading errors. While explicit 3D grounding stabilizes reasoning on depth-sensitive tasks, imperfect 3D representation proves more harmful than 2D baselines by distorting spatial relations. Human studies further reveal that unlike humans who seek falsifying viewpoints and revise beliefs under contradiction, models commit prematurely with high confidence regardless of evidence quality, exposing a metacognitive gap that neither better perception nor more embodied interaction alone can close.

Related papers

• MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents [74.83535434786145]
  MNIST-PRO is a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints.<n>We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas.
  arXiv  Detail & Related papers  (2026-08-31T16:05:39Z)
• Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction [39.71864879537297]
  ZendoWorld is a controlled interactive environment in which agents must infer a logical rule about visual game observations.<n>We evaluate several agents spanning pure VLM reasoning, Bayesian particle filtering, dynamic concept discovery, and neuro-symbolic methods.<n>Our main findings are: (1) high accuracy in predicting labels does not imply recovery of the underlying rule; (2) perception and induction are distinct bottlenecks for different agent classes; and (3) VLM-based agents propose near-uninformative experiments, failing to actively reduce hypothesis uncertainty.
  arXiv  Detail & Related papers  (2026-07-09T08:29:23Z)
• SpatialAct: Probing Spatial Reasoning-to-Action Capabilities of VLM Agents in 3D Scenes [15.076506371158876]
  Humans can effortlessly perceive spatial layouts, form cognitive representations, and translate such reasoning into actions in everyday 3D environments.<n>Recent vision-language models (VLMs) have shown promising performance on observation-conditioned spatial perception and reasoning tasks.<n>We introduce textbfSpatialAct, a simulator-grounded benchmark for probing textitaction-conditioned spatial reasoning in 3D scenes.
  arXiv  Detail & Related papers  (2026-05-29T10:59:26Z)
• The Bystander Effect in Multi-Agent Reasoning: Quantifying Cognitive Loafing in Collaborative Interactions [6.357772907811544]
  We evaluate 22,500 deterministic trajectories across 3 dataset contexts with 3 state-of-the-art (SOTA) models.<n>We formalize the textitSovereignty Gap: models frequently compute the correct derivation internally but suffer Alignment Hallucinations''<n>We prove that multi-agent social load is strictly non-commutative; the "brand" identity of the Lead Anchor'' auditor disproportionately dictates the swarm's integrity.
  arXiv  Detail & Related papers  (2026-05-11T15:13:01Z)
• How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Study [17.830219852925715]
  We focus on a fundamental and critical capability in spatial intelligence from a linguistic perspective: viewpoint rotation understanding (VRU)<n>We find that both LLMs and VLMs perform poorly on our proposed dataset while human can easily achieve 100% accuracy.<n>Our findings reveal that although models encode viewpoint information in the hidden states, they appear to struggle to bind the viewpoint position with corresponding observation.
  arXiv  Detail & Related papers  (2026-04-16T17:55:19Z)
• Cognitive Pivot Points and Visual Anchoring: Unveiling and Rectifying Hallucinations in Multimodal Reasoning Models [65.57985131861399]
  hallucinations are strongly correlated with cognitive bifurcation points that often exhibit high entropy states.<n>We propose V-STAR, a lightweight, holistic training paradigm designed to internalize visually aware reasoning capabilities.
  arXiv  Detail & Related papers  (2026-04-11T13:59:05Z)
• Attention at Rest Stays at Rest: Breaking Visual Inertia for Cognitive Hallucination Mitigation [50.51650162235191]
  We propose a training-free Inertia-aware Visual Excitation (IVE) method that breaks this inertial pattern by modeling cognitive inference as the dynamic responsiveness of visual attention.<n>IVE is effective across various base MLLMs and multiple hallucination benchmarks, particularly for cognitive hallucinations.
  arXiv  Detail & Related papers  (2026-04-02T12:51:07Z)
• Anatomy of a Lie: A Multi-Stage Diagnostic Framework for Tracing Hallucinations in Vision-Language Models [62.932580559941414]
  Vision-Language Models (VLMs) frequently "hallucinate" - generate plausible yet factually incorrect statements.<n>We propose a new paradigm for diagnosing hallucinations, recasting them from static output errors into dynamic pathologies of a model's computational cognition.
  arXiv  Detail & Related papers  (2026-03-16T17:20:38Z)
• Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification [27.02252748004729]
  Large vision-language models (LVLMs) have shown substantial advances in multimodal understanding and generation.<n>They frequently produce unreliable or even harmful content, such as fact hallucinations or dangerous instructions.<n>Evidential Uncertainty Quantification (EUQ) captures both information conflict and ignorance for effective detection of LVLM misbehaviors.
  arXiv  Detail & Related papers  (2026-02-05T10:51:39Z)
• Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration? [83.13508919229939]
  Theory of Space is defined as an agent's ability to actively acquire information through self-directed, active exploration.<n>A key innovation is spatial belief probing, which prompts models to reveal their internal spatial representations at each step.<n>Our findings suggest that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration.
  arXiv  Detail & Related papers  (2026-02-04T19:06:40Z)
• HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models [27.72821031361892]
  We present the first systematic study of hallucinations in large language models performing long-horizon tasks under scene-task inconsistencies.<n>Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond.
  arXiv  Detail & Related papers  (2025-06-18T02:13:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.