Beyond Sally-Anne: Evaluating Theory of Mind in LLMs using Epistemic Schelling Points

• URL: http://arxiv.org/abs/2607.11363v1
• Date: Mon, 13 Jul 2026 10:27:22 GMT
• Title: Beyond Sally-Anne: Evaluating Theory of Mind in LLMs using Epistemic Schelling Points
• Abstract summary: We introduce the Epistemic Asymmetry Schelling Task (EAST), a two-player dialogue game designed to benchmark robust ToM abilities.<n>We evaluate whether models can robustly apply ToM to achieve coordination.<n>Our results reveal a significant capability gap in functional social reasoning.
• Score: 0.8703469130213444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-based evaluations of Theory of Mind (ToM) in Large Language Models (LLMs) often involve cognitive tests akin to the Sally-Anne task that can be gamed due to exposure to relevantly similar tasks in pre-training and do not obviously test models' functional ToM abilities in ways that generalize to naturalistic settings. To address these issues, we introduce the Epistemic Asymmetry Schelling Task (EAST), a two-player dialogue game designed to benchmark robust and generalizable ToM abilities. By requiring LLM-LLM dyads to independently converge on semantic Schelling points under varying states of epistemic transparency, we evaluate whether models can robustly apply ToM to achieve coordination. Our results reveal a significant capability gap in functional social reasoning, with only frontier models successfully navigating the varying epistemic demands of the tasks. Analysis of reasoning traces shows that coordination failures are primarily driven by epistemic tracking errors, such as conflating private knowledge with mutual knowledge. Despite high performance on traditional static benchmarks, our study shows that robust social reasoning and epistemic tracking remain a critical bottleneck, providing concrete targets for future LLM evaluation and development.

Related papers

• Compliance versus Sensibility: On the Reasoning Controllability in Large Language Models [46.26628756478016]
  Large Language Models (LLMs) are known to acquire reasoning capabilities through shared inference patterns in pre-training data.<n>We show that reasoning conflicts are internally detectable, as confidence scores significantly drop during conflicting episodes.<n>Our findings establish that while LLM reasoning is anchored to concrete instances, active mechanistic interventions can effectively decouple logical schemata from data.
  arXiv  Detail & Related papers  (2026-04-29T22:55:40Z)
• Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind [8.740788873949471]
  Large Language Models (LLMs) have developed rapidly and are widely applied to both general-purpose and professional tasks.<n>They still struggle to comprehend and respond to the true user needs when intentions and instructions are imprecisely conveyed.
  arXiv  Detail & Related papers  (2026-02-14T16:01:59Z)
• Reasoning Promotes Robustness in Theory of Mind Tasks [0.26945563448932225]
  Large language models (LLMs) have recently shown strong performance on Theory of Mind (ToM) tests.<n>This paper examines the behavior of such reasoning models in ToM tasks using novel adaptations of machine psychological experiments and results from established benchmarks.
  arXiv  Detail & Related papers  (2026-01-23T16:01:24Z)
• LLM-MC-Affect: LLM-Based Monte Carlo Modeling of Affective Trajectories and Latent Ambiguity for Interpersonal Dynamic Insight [1.1119672724275114]
  Emotional coordination is a core property of human interaction that shapes how meaning is constructed in real time.<n>We introduce a probabilistic framework that characterizes emotion not as a static label, but as a continuous latent probability distribution.<n>This work establishes a scalable and deployable pathway for understanding interpersonal dynamics, offering a generalizable solution.
  arXiv  Detail & Related papers  (2026-01-07T06:50:41Z)
• How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns [51.02752099869218]
  Large Language Models (LLMs) display strikingly different generalization behaviors.<n>We introduce a novel benchmark that decomposes reasoning into atomic core skills.<n>We show that RL-tuned models maintain more stable behavioral profiles and resist collapse in reasoning skills, whereas SFT models exhibit sharper drift and overfit to surface patterns.
  arXiv  Detail & Related papers  (2025-12-30T08:16:20Z)
• MM-OPERA: Benchmarking Open-ended Association Reasoning for Large Vision-Language Models [15.929002709503921]
  We aim to evaluate a fundamental yet underexplored intelligence: association.<n> MM-OPERA is a systematic benchmark with 11,497 instances across two open-ended tasks.<n>It challenges LVLMs to resemble the spirit of divergent thinking and convergent associative reasoning.
  arXiv  Detail & Related papers  (2025-10-30T18:49:06Z)
• How Good are Foundation Models in Step-by-Step Embodied Reasoning? [79.15268080287505]
  Embodied agents must make decisions that are safe, spatially coherent, and grounded in context.<n>Recent advances in large multimodal models have shown promising capabilities in visual understanding and language generation.<n>Our benchmark includes over 1.1k samples with detailed step-by-step reasoning across 10 tasks and 8 embodiments.
  arXiv  Detail & Related papers  (2025-09-18T17:56:30Z)
• CTRLS: Chain-of-Thought Reasoning via Latent State-Transition [57.51370433303236]
  Chain-of-thought (CoT) reasoning enables large language models to break down complex problems into interpretable intermediate steps.<n>We introduce groundingS, a framework that formulates CoT reasoning as a Markov decision process (MDP) with latent state transitions.<n>We show improvements in reasoning accuracy, diversity, and exploration efficiency across benchmark reasoning tasks.
  arXiv  Detail & Related papers  (2025-07-10T21:32:18Z)
• Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation [106.17986469245302]
  Large language models (LLMs) have demonstrated impressive reasoning capacities that mirror human-like thinking.<n>Existing reasoning benchmarks either focus on domain-specific knowledge (crystallized intelligence) or lack interpretability.<n>We propose DRE-Bench, a dynamic reasoning evaluation benchmark grounded in a hierarchical cognitive framework.
  arXiv  Detail & Related papers  (2025-06-03T09:01:08Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation [60.65820977963331]
  We introduce a novel evaluation paradigm for Large Language Models (LLMs) This paradigm shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-12-28T15:49:43Z)
• Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models [61.28463542324576]
  Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can generate human-like outputs. We evaluate existing state-of-the-art VLMs and find that even the best-performing model is unable to demonstrate strong visual reasoning capabilities and consistency. We propose a two-stage training framework aimed at improving both the reasoning performance and consistency of VLMs.
  arXiv  Detail & Related papers  (2023-09-08T17:49:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.