EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents

• URL: http://arxiv.org/abs/2605.09826v2
• Date: Fri, 15 May 2026 19:33:36 GMT
• Title: EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents
• Abstract summary: Existing benchmarks mostly test literal Theory of Mind (ToM) by asking direct belief questions.<n>We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household.<n>On the hard split, all seven evaluated frontier models score 0.0% Pass3 on functional task completion, while averaging 45.0% on literal belief probes.
• Score: 75.01735520608075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Theory of Mind (ToM), the ability to track others epistemic state, makes humans efficient collaborators. AI agents need the same capacity in multi agent settings, yet existing benchmarks mostly test literal ToM by asking direct belief questions. The ability act optimally on implicit beliefs in embodied environments, called functional ToM, remains largely untested. We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household with partial observability, private information, and constrained communication. Each task is formally verified for solvability and required epistemic depth, and new tasks are generated increase difficulty as models improve. On the hard split, all seven evaluated frontier models score 0.0% Pass^3 on functional task completion, while averaging 45.0% on literal belief probes. Manual analysis traces 93% of sampled failures to epistemic coordination breakdowns such as withheld information, ignored partner constraints, and misallocated messages, providing a concrete target for future work.

Related papers

• Efficient Test-Time Adaptation through Human-AI Interaction [112.5131603022122]
  We propose test-time adaptation through human-agent interaction (TAHI)<n>We adapt agents to 30 individuals in two high-utility domains, writing and visual creation, on a total of 600 tasks.<n>Our agents improve solo task success by 4.5-20.9% within only tens of tasks.
  arXiv  Detail & Related papers  (2026-09-03T17:33:18Z)
• SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information [61.41881137371855]
  We introduce SPIEval, a benchmark for large language models (LLMs)<n> SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools.<n>The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%.
  arXiv  Detail & Related papers  (2026-08-11T09:14:53Z)
• Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action [3.236973203905278]
  We evaluate an agent's ability to induce specific belief states in other agents by taking actions rather than using conversational persuasion.<n>GPT-5 was successful on approximately 80% of tasks in the agentic setting.<n>All models, like humans, performed better on tasks inducing true belief states than false belief states.
  arXiv  Detail & Related papers  (2026-06-30T16:22:12Z)
• SpaceMind: A Modular and Self-Evolving Embodied Vision-Language Agent Framework for Autonomous On-orbit Servicing [1.614712872796152]
  SpaceMind is a self-evolving vision-language model that decomposes knowledge, tools, and reasoning into three dimensions.<n>We validate SpaceMind through 192 closed-loop runs across five satellites, three task types, and two environments.<n>A self-evolution study shows that the agent recovers from failure in four of six groups from a single failed episode.
  arXiv  Detail & Related papers  (2026-04-15T20:27:57Z)
• Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence? [35.30497528897595]
  Agentic-MME is a process-verified benchmark for Multimodal Agentic Capabilities.<n>It contains 418 real-world tasks across 6 domains and 3 difficulty levels.<n>It features over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task.
  arXiv  Detail & Related papers  (2026-04-03T13:02:01Z)
• Automatic Cognitive Task Generation for In-Situ Evaluation of Embodied Agents [43.01384379901339]
  We propose a dynamic in-situ task generation method for unseen environments inspired by human cognition.<n>In the interaction stage, the agent actively interacts with the environment, creating a loop between task execution and generation.<n>Experiments across 10 unseen scenes demonstrate that TEA automatically generated 87,876 tasks in two cycles.
  arXiv  Detail & Related papers  (2026-02-05T03:07:00Z)
• Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces [126.23612941699565]
  Terminal-Bench 2.0 is a benchmark composed of 89 tasks in computer terminal environments inspired by problems from real world.<n>We show that frontier models and agents score less than 65% on the benchmark.<n>We publish the dataset and evaluation harness to assist developers and researchers in future work at https://www.tbench.ai/.
  arXiv  Detail & Related papers  (2026-01-17T01:29:30Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents [12.083458582842384]
  This work focuses on creating WAGIBench, a strong benchmark to measure progress in solving this problem using vision-language models (VLMs)<n>We collect 29 hours of multimodal data from 348 participants across 3,477 recordings, featuring ground-truth goals alongside accompanying visual, audio, digital, and longitudinal contextual observations.<n>We validate that human performance exceeds model performance, achieving 93% multiple-choice accuracy compared with 84% for the best-performing VLM.
  arXiv  Detail & Related papers  (2025-10-25T21:54:01Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• OmniEAR: Benchmarking Agent Reasoning in Embodied Tasks [52.87238755666243]
  We present OmniEAR, a framework for evaluating how language models reason about physical interactions, tool usage, and multi-agent coordination in embodied tasks.<n>We model continuous physical properties and complex spatial relationships across 1,500 scenarios spanning household and industrial domains.<n>Our systematic evaluation reveals severe performance degradation when models must reason from constraints.
  arXiv  Detail & Related papers  (2025-08-07T17:54:15Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.