OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling

• URL: http://arxiv.org/abs/2605.26322v1
• Date: Mon, 25 May 2026 20:45:08 GMT
• Title: OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling
• Abstract summary: Theory of Mind (ToM) is commonly evaluated in large language models (LLMs) using end-point question answering.<n>This paradigm obscures whether the model actually constructs the underlying mental-state representations required for robust reasoning.<n>We introduce OmniToM, a benchmark that directly evaluates these representations by requiring explicit modeling of belief structures for all relevant actors within a narrative.
• Score: 49.9159527733006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Theory of Mind (ToM), the ability to infer others' knowledge, intentions, and emotions, is commonly evaluated in large language models (LLMs) using end-point question answering, where performance is judged solely by the final answer to a social reasoning query. This paradigm obscures whether the model actually constructs the underlying mental-state representations required for robust reasoning, particularly in scenarios involving divergent, evolving, or mistaken beliefs. In order to address this research gap, we introduce OmniToM, a benchmark that directly evaluates these representations by requiring explicit modeling of belief structures for all relevant actors within a narrative. These structures are composed of belief propositions: minimal statements of what an actor takes to be true about the world or another actor's mental state, allowing knowledge, intentions, emotions, and false beliefs to be analyzed in a common format. Models are evaluated in two stages: Stage 1: Belief Extraction, which extracts from the story the beliefs relevant to its social dynamics, and Stage 2: Belief Labeling, which assigns each belief a seven-dimensional schema label covering recursive order, truth status, knowledge access, explicitness, content type, mental source, and context. Built from 895 stories from the existing ToMBench story corpus and augmented with 22,343 labeled belief propositions, OmniToM uses a human-calibrated LLM-assisted annotation pipeline. Across diverse models in zero-shot evaluation, OmniToM reveals an actor-specific belief-tracking bottleneck: current LLMs struggle with the knowledge-access and representational decisions required to transform narrative facts into actors' beliefs and shared mental states.

Related papers

• Dynamic Theory of Mind as a Temporal Memory Problem: Evidence from Large Language Models [0.8666275811953881]
  Theory of Mind (ToM) is central to social cognition and human-AI interaction.<n>Large Language Models (LLMs) have been used to help understand and represent ToM.<n>We introduce DToM-Track to investigate temporal belief reasoning in controlled multiturn conversations.
  arXiv  Detail & Related papers  (2026-03-15T22:54:03Z)
• TactfulToM: Do LLMs Have the Theory of Mind Ability to Understand White Lies? [13.075782848287487]
  We introduce TactfulToM, a novel English benchmark designed to evaluate Large Language Models' (LLMs) ability to understand white lies within real-life conversations.<n>Our benchmark is generated through a multi-stage human-in-the-loop pipeline where LLMs expand manually designed seed stories into conversations to maintain the information asymmetry necessary for authentic white lies.<n>We show that TactfulToM is challenging for state-of-the-art models, which perform substantially below humans, revealing shortcomings in their ability to fully comprehend the ToM reasoning that enables true understanding of white lies.
  arXiv  Detail & Related papers  (2025-09-21T12:18:35Z)
• Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding [97.05584099530226]
  We introduce MF$2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies.<n>For each pair, models must correctly identify both the true and false claims.<n>Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance.
  arXiv  Detail & Related papers  (2025-06-06T17:58:36Z)
• How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognition [75.11808682808065]
  This study investigates whether large language models (LLMs) exhibit similar tendencies in understanding semantic size.<n>Our findings reveal that multi-modal training is crucial for LLMs to achieve more human-like understanding.<n> Lastly, we examine whether LLMs are influenced by attention-grabbing headlines with larger semantic sizes in a real-world web shopping scenario.
  arXiv  Detail & Related papers  (2025-03-01T03:35:56Z)
• ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind [25.524355451378593]
  ToMATO is a new ToM benchmark formulated as multiple-choice QA over conversations.<n>We capture both first- and second-order mental states across five categories: belief, intention, desire, emotion, and knowledge.<n>ToMATO consists of 5.4k questions, 753 conversations, and 15 personality trait patterns.
  arXiv  Detail & Related papers  (2025-01-15T14:47:02Z)
• The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters [67.61587661660852]
  Theory-of-Mind (ToM) allows humans to understand and interpret the mental states of others.<n>In this paper, we verify the importance of comprehensive contextual understanding about personal backgrounds in ToM.<n>We introduce CharToM benchmark, comprising 1,035 ToM questions based on characters from classic novels.
  arXiv  Detail & Related papers  (2025-01-03T09:04:45Z)
• Understanding Epistemic Language with a Language-augmented Bayesian Theory of Mind [47.001163099930494]
  We introduce a cognitive model of epistemic language interpretation grounded in Bayesian inferences.<n>We validate our model in an experiment where participants watch an agent navigate a maze to find keys hidden in boxes needed to reach their goal, then rate sentences about the agent's beliefs.
  arXiv  Detail & Related papers  (2024-08-21T22:29:56Z)
• Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models [51.91448005607405]
  We evaluate key human ToM precursors by annotating characters' perceptions on ToMi and FANToM. We present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference.
  arXiv  Detail & Related papers  (2024-07-08T14:58:29Z)
• How FaR Are Large Language Models From Agents with Theory-of-Mind? [69.41586417697732]
  We propose a new evaluation paradigm for large language models (LLMs): Thinking for Doing (T4D) T4D requires models to connect inferences about others' mental states to actions in social scenarios. We introduce a zero-shot prompting framework, Foresee and Reflect (FaR), which provides a reasoning structure that encourages LLMs to anticipate future challenges.
  arXiv  Detail & Related papers  (2023-10-04T06:47:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.