SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

• URL: http://arxiv.org/abs/2606.09669v1
• Date: Mon, 08 Jun 2026 15:51:51 GMT
• Title: SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks
• Abstract summary: SpatialWorld is a benchmark for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks.<n>It features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration)<n>For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier.
• Score: 73.92333717662558
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However, existing benchmarks predominantly rely on passive evaluation (e.g., static VQA) or simulator-specific pipelines, failing to assess general interactive spatial understanding. We introduce SpatialWorld, a unified benchmark designed specifically for evaluating the interactive spatial understanding of multimodal agents in complex real-world tasks. Integrating eight heterogeneous simulation backends under a shared, simulator-agnostic protocol, SpatialWorld features 760 human-annotated tasks across diverse domains (e.g., household routines, travel, social collaboration). Agents must solve tasks under vision-only partial observability, actively gathering egocentric visual evidence and expressing decisions via a unified, text-based action interface native to MLLMs. For reliable evaluation, each task includes a human-validated initial state, a reference trajectory, and a terminal-state verifier. Evaluating 15 advanced agents reveals that robust spatial task solving remains challenging: the strongest model, GPT-5, achieves an average task success rate (TSR) of only 17.4%, while the leading open-source model, Qwen-3.5, reaches 14.1%. Further analysis exposes a clear mismatch between task success and execution efficiency, alongside substantial domain-specific performance variations. These bottlenecks in active exploration and long-horizon planning position SpatialWorld as a rigorous testbed for future spatial agents.

Related papers

• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• Benchmarking General Mobile Assistants in Challenging Real-World Scenarios [30.863445606315874]
  We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios.<n>GMA introduces seven applications based on open-source projects, spanning domains such as lifestyle sharing and travel planning, and 300 tasks across four difficulty tiers.<n>We find that performance declines substantially as task complexity increases, with current agents remaining far from reliably handling realistic user requirements.
  arXiv  Detail & Related papers  (2026-08-21T12:17:14Z)
• Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation [47.58516950734307]
  Existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges.<n>We present REAL, an agentic framework for open-world mobile manipulation.<n>We design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning.
  arXiv  Detail & Related papers  (2026-07-15T09:55:45Z)
• OccuBench: Evaluating AI Agents on Real-World Professional Tasks via Language Environment Simulation [57.505743202759646]
  OccuBench is a benchmark covering 100 real-world professional task scenarios across 10 industry categories and 65 specialized domains.<n>Our multi-agent synthesis pipeline automatically produces evaluation instances with guaranteed solvability, calibrated difficulty, and document-grounded diversity.
  arXiv  Detail & Related papers  (2026-04-13T00:27:32Z)
• ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation [66.02142169323521]
  Vision-Language-Action models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence.<n>Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise.<n>We introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution.
  arXiv  Detail & Related papers  (2026-03-30T15:06:41Z)
• TongSIM: A General Platform for Simulating Intelligent Machines [59.27575233453533]
  Embodied intelligence focuses on training agents within realistic simulated environments.<n>TongSIM is a high-fidelity, general-purpose platform for training and evaluating embodied agents.
  arXiv  Detail & Related papers  (2025-12-23T10:00:43Z)
• BEAR: Benchmarking and Enhancing Multimodal Language Models for Atomic Embodied Capabilities [61.173773299032746]
  Embodied capabilities refer to a suite of fundamental abilities for an agent to perceive, comprehend, and interact with the physical world.<n>We introduce BEAR, a comprehensive and fine-grained benchmark that evaluates MLLMs on atomic embodied capabilities.<n> BEAR comprises 4,469 interleaved image-video-text entries across 14 domains in 6 categories, including tasks from low-level pointing, trajectory understanding, spatial reasoning, to high-level planning.<n>We propose BEAR-Agent, a multimodal conversable agent that integrates pretrained vision models to strengthen MLLM perception, 3D understanding, and planning capabilities.
  arXiv  Detail & Related papers  (2025-10-09T19:18:36Z)
• REVELIO -- Universal Multimodal Task Load Estimation for Cross-Domain Generalization [2.689067085628911]
  This paper introduces a new multimodal dataset that extends established cognitive load detection benchmarks with a real-world gaming application.<n>Task load annotations are derived from objective performance, subjective NASA-TLX ratings, and task-level design.<n>State-of-the-art end-to-end model, including xLSTM, ConvNeXt, and Transformer architectures are systematically trained and evaluated.
  arXiv  Detail & Related papers  (2025-09-01T17:36:09Z)
• Holistic Evaluation of Multimodal LLMs on Spatial Intelligence [81.2547965083228]
  We propose EASI for holistic Evaluation of multimodAl LLMs on Spatial Intelligence.<n>We conduct the study across eight key benchmarks, at a cost exceeding ten billion total tokens.<n>Our empirical study then reveals that GPT-5 demonstrates unprecedented strength in spatial intelligence (SI), yet (2) still falls short of human performance significantly across a broad spectrum of SI-tasks.
  arXiv  Detail & Related papers  (2025-08-18T17:55:17Z)
• StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley [17.706348191734822]
  We introduce StarDojo, a novel benchmark to assess AI agents in open-ended production-living simulations.<n>StarDojo features 1,000 meticulously curated tasks across five key domains: farming, crafting, exploration, combat, and social interactions.<n>We provide a compact subset of 100 representative tasks for efficient model evaluation.
  arXiv  Detail & Related papers  (2025-07-10T05:48:28Z)
• REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites [9.58858258192147]
  We introduce REAL, a benchmark and framework for multi-turn agent evaluations on deterministic simulations of real-world websites.<n>We also release a benchmark consisting of 112 practical tasks that mirror everyday complex user interactions.<n>Our framework supports easy integration of new tasks, reproducible evaluation, and scalable post-training data generation.
  arXiv  Detail & Related papers  (2025-04-15T18:22:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.