ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions

• URL: http://arxiv.org/abs/2509.05066v2
• Date: Tue, 16 Sep 2025 12:22:34 GMT
• Title: ToM-SSI: Evaluating Theory of Mind in Situated Social Interactions
• Abstract summary: ToM-SSI is a new benchmark specifically designed to test ToM capabilities in environments rich with social interactions and spatial dynamics.<n>This unique design allows us to study, for the first time, mixed cooperative-obstructive settings and reasoning about multiple agents' mental state in parallel.
• Score: 17.2191603793837
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Most existing Theory of Mind (ToM) benchmarks for foundation models rely on variations of the Sally-Anne test, offering only a very limited perspective on ToM and neglecting the complexity of human social interactions. To address this gap, we propose ToM-SSI: a new benchmark specifically designed to test ToM capabilities in environments rich with social interactions and spatial dynamics. While current ToM benchmarks are limited to text-only or dyadic interactions, ToM-SSI is multimodal and includes group interactions of up to four agents that communicate and move in situated environments. This unique design allows us to study, for the first time, mixed cooperative-obstructive settings and reasoning about multiple agents' mental state in parallel, thus capturing a wider range of social cognition than existing benchmarks. Our evaluations reveal that the current models' performance is still severely limited, especially in these new tasks, highlighting critical gaps for future research.

Related papers

• MeetingToM: Evaluating Multimodal LLMs on Theory-of-Mind Reasoning in Multi-Party Meetings [11.551333946022053]
  We introduce MeetingToM, a benchmark for complex social behavior reasoning in naturalistic multi-party meetings.<n>The benchmark is hierarchically organized to evaluate ToM at increasing levels of social granularity.<n>Our results highlight key challenges and establish MeetingToM as a testbed for advancing meeting-grounded ToM in multimodal models.
  arXiv  Detail & Related papers  (2026-07-21T16:05:49Z)
• Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations [80.39504840645687]
  Existing benchmarks measure ToM capability improvement through story-reading, multiple-choice questions from a third-person perspective.<n>This paper proposes a new paradigm of interactive ToM evaluation with both perspective and metric shifts.<n>Our findings reveal that improvements on static benchmarks do not always translate to better performance in dynamic HAI interactions.
  arXiv  Detail & Related papers  (2026-04-28T15:38:31Z)
• SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models [86.19617358080016]
  Social Omni is a benchmark that operationalizes the evaluation of conversational interactivity across three core dimensions.<n>Social Omni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances.<n>Our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions.
  arXiv  Detail & Related papers  (2026-03-17T17:58:44Z)
• CoMMET: To What Extent Can LLMs Perform Theory of Mind Tasks? [26.35781229730513]
  Theory of Mind (ToM)-the ability to reason about the mental states of oneself and others-is a cornerstone of human social intelligence.<n>Existing benchmarks for assessing ToM in Large Language Models (LLMs) are limited.
  arXiv  Detail & Related papers  (2026-03-12T13:29:42Z)
• The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation [0.16921396880325779]
  We introduce a novel evaluation framework that uses multi-agent debate as a controlled "social laboratory"<n>We show that assigned personas induce stable, measurable psychometric profiles, particularly in cognitive effort.<n>This work provides a blueprint for a new class of dynamic, psychometrically grounded evaluation protocols.
  arXiv  Detail & Related papers  (2025-10-01T07:10:28Z)
• Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner [32.33827730707331]
  We propose a scalable Bayesian ToM planner that decomposes ToM reasoning into stepwise Bayesian updates.<n>Our framework introduces weak-to-strong control, allowing smaller language models to specialize in ToM-specific likelihood estimation.<n>Our method achieves a 4.6% accuracy improvement over state-of-the-art techniques on multimodal ToM benchmarks.
  arXiv  Detail & Related papers  (2025-06-02T04:23:45Z)
• Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective [24.27038998164743]
  Theory-of-Mind (ToM) tasks are designed for humans to benchmark LLM's ToM capabilities.<n>This approach has a number of limitations.<n>Taking a human-computer interaction (HCI) perspective, these limitations prompt us to rethink the definition and criteria of ToM in ToM benchmarks.
  arXiv  Detail & Related papers  (2025-04-15T03:44:43Z)
• NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding [55.38254464415964]
  Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states.
  arXiv  Detail & Related papers  (2024-04-21T11:51:13Z)
• AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents [65.16893197330589]
  Large Language Models (LLMs) have demonstrated their ability to replicate human behaviors across a wide range of scenarios. However, their capability in handling complex, multi-character social interactions has yet to be fully explored. We introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods.
  arXiv  Detail & Related papers  (2024-01-12T11:18:00Z)
• Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities [63.90227161974381]
  SimToM is a novel prompting framework inspired by Simulation Theory's notion of perspective-taking. Our approach, which requires no additional training and minimal prompt-tuning, shows substantial improvement over existing methods.
  arXiv  Detail & Related papers  (2023-11-16T22:49:27Z)
• FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions [94.61530480991627]
  Theory of mind evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering.
  arXiv  Detail & Related papers  (2023-10-24T00:24:11Z)
• Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View [60.80731090755224]
  This paper probes the collaboration mechanisms among contemporary NLP systems by practical experiments with theoretical insights. We fabricate four unique societies' comprised of LLM agents, where each agent is characterized by a specific trait' (easy-going or overconfident) and engages in collaboration with a distinct thinking pattern' (debate or reflection) Our results further illustrate that LLM agents manifest human-like social behaviors, such as conformity and consensus reaching, mirroring social psychology theories.
  arXiv  Detail & Related papers  (2023-10-03T15:05:52Z)
• Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models [82.50173296858377]
  Many anecdotal examples were used to suggest newer large language models (LLMs) like ChatGPT and GPT-4 exhibit Neural Theory-of-Mind (N-ToM) We investigate the extent of LLMs' N-ToM through an extensive evaluation on 6 tasks and find that while LLMs exhibit certain N-ToM abilities, this behavior is far from being robust.
  arXiv  Detail & Related papers  (2023-05-24T06:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.