MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents

• URL: http://arxiv.org/abs/2608.07533v1
• Date: Sun, 26 Jul 2026 14:07:20 GMT
• Title: MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents
• Abstract summary: MetaSpace is a novel framework designed to evaluate the spatial cognition of embodied agents.<n>We show that MetaSpace successfully detects 90,422 spatial cognition errors in state-of-the-art (SOTA) MLLM-driven agents.<n>Results indicate that all SOTA agents achieve average scores between 0.44 and 0.52, significantly lower than the human benchmark of 0.96.
• Score: 7.0922719251235655
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An embodied agent is an intelligent entity that interacts with its environment through a physical body. Currently, the evaluation of embodied agents primarily relies on two paradigms: (1) manually annotated Visual Question Answering (VQA) pairs and (2) high-level task completion metrics, such as success in navigation or manipulation. The former is labor-intensive and subject to variability in annotation quality. The latter may obscure critical vulnerabilities, allowing agents to complete tasks through suboptimal means or safety violations, thereby concealing safety risks and inefficiencies. Given that spatial cognition is the cornerstone for executing embodied tasks, there is a pressing need to assess whether embodied agents possess robust spatial cognition during task execution. Inspired by metamorphic testing principles in software engineering, we propose MetaSpace, a novel framework designed to evaluate the spatial cognition of agents. By leveraging spatiotemporal multimodal states derived from real execution trajectories, MetaSpace automatically generates test cases based on predefined metamorphic relations (MRs) grounded in logical rules and physical laws. Crucially, we encode these MRs as executable rules in a logic programming language (Prolog). Violations of these relations indicate failures in spatial cognition. Our empirical evaluation across three embodied scenarios demonstrates that MetaSpace successfully detects 90,422 spatial cognition errors in state-of-the-art (SOTA) MLLM-driven agents. We introduce the Spatial Cognition (SC) score to quantify performance. Results indicate that all SOTA agents achieve average scores between 0.44 and 0.52, significantly lower than the human benchmark of 0.96.

Related papers

• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development? [80.24951682268332]
  We introduce the Meta-Agent Challenge (MAC), an evaluation framework designed to test the capacity of frontier models for autonomous agent development.<n>To ensure evaluation integrity, this framework is secured by multi-layer defenses against reward hacking.<n>We demonstrate that meta-agents rarely match human-engineered baseline policies, and the few that do are dominated by proprietary frontier models.
  arXiv  Detail & Related papers  (2026-06-03T04:58:17Z)
• Beyond Binary Success: A Diagnostic Meta-Evaluation Framework for Fine-Grained Manipulation [98.79811866787263]
  We introduce MetaFine, a diagnostic meta-evaluation framework that disentangles manipulation competency along three axes.<n>We identify the visual encoder's ability to preserve local spatial structure as a key bottleneck for fine-grained precision.<n>By shifting evaluation from ranking to diagnosis, MetaFine turns benchmarking into an actionable compass for repairing the layered capacities underlying genuine physical dexterity.
  arXiv  Detail & Related papers  (2026-05-19T15:25:13Z)
• Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks [16.03812136589077]
  We introduce an Anchor-Based Embodied Spatial Decomposition Chain-of-Thought (CoT)<n>CoT guides the MLLM through a "geometric-to-semantic" projection, forcing it to first establish B's local coordinate system and then dynamically weight visual and auditory modalities based on whether A falls within B's visual frustum.
  arXiv  Detail & Related papers  (2026-05-18T10:32:56Z)
• SAGE: A Service Agent Graph-guided Evaluation Benchmark [27.342044311161654]
  We propose SAGE (Service Agent Graph-guided Evaluation), a universal benchmark for automated, dual-axis assessment.<n>SAGE formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise verification of logical compliance.<n>We also observe Empathy Resilience'', a phenomenon where models maintain polite conversational facades despite underlying logical failures.
  arXiv  Detail & Related papers  (2026-04-10T12:55:23Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild [30.138230316314534]
  We introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction to bidirectional intent alignment.<n>We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols.<n>We also develop MUSE, an automated framework utilizing an MLLM-as-a-judge multi-agent architecture.
  arXiv  Detail & Related papers  (2026-02-12T09:25:15Z)
• OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows [9.617220633655716]
  We present textbfunderlineOmni-textbfunderlineModality textbfunderlineGeneration Agent (textbfOMG-Agent)
  arXiv  Detail & Related papers  (2026-02-04T02:25:40Z)
• What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding [50.35012849818872]
  Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks.<n>We propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding.<n>Our experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment.
  arXiv  Detail & Related papers  (2026-01-14T14:09:11Z)
• Agent4FaceForgery: Multi-Agent LLM Framework for Realistic Face Forgery Detection [108.5042835056188]
  This work introduces Agent4FaceForgery to address two fundamental problems.<n>How to capture the diverse intents and iterative processes of human forgery creation.<n>How to model the complex, often adversarial, text-image interactions that accompany forgeries in social media.
  arXiv  Detail & Related papers  (2025-09-16T01:05:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.