DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories

• URL: http://arxiv.org/abs/2604.20443v1
• Date: Wed, 22 Apr 2026 11:07:46 GMT
• Title: DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories
• Abstract summary: We introduce DialToM, a human-verified benchmark built from natural human dialogue.<n>We evaluate whether models can identify state-consistent dialogue trajectories solely from mental-state profiles.
• Score: 21.300337724962336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have been shown to possess Theory of Mind (ToM) abilities. However, it remains unclear whether this stems from robust reasoning or spurious correlations. We introduce DialToM, a human-verified benchmark built from natural human dialogue using a multiple-choice framework. We evaluate not only mental state prediction (Literal ToM) but also the functional utility of these states (Functional ToM) through Prospective Diagnostic Forecasting -- probing whether models can identify state-consistent dialogue trajectories solely from mental-state profiles. Our results reveal a significant reasoning asymmetry: while LLMs excel at identifying mental states, most (except for Gemini 3 Pro) fail to leverage this understanding to forecast social trajectories. Additionally, we find only weak semantic similarities between human and LLM-generated inferences. To facilitate reproducibility, the DialToM dataset and evaluation code are publicly available at https://github.com/Stealth-py/DialToM.

Related papers

• Decoupling Turn-Taking from Semantics: A Decoupled Data Approach for Finite-State-Machine-Based Full-Duplex Dialogue [23.813275217960093]
  Large Models (LLMs) cannot faithfully simulate the fine-grained acoustic temporal dynamics of real human dialogues.<n>We propose a decoupled data approach that learns turn-taking from real Human-Human (HH) spoken dialogues while shaping behavior through semantic HumanAgent (HA) text dialogues.
  arXiv  Detail & Related papers  (2026-09-03T03:17:07Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• GPT-4o Lacks Core Features of Theory of Mind [0.09320657506524145]
  We use a cognitively-grounded definition of ToM to develop and test a new evaluation framework.<n>We find that even though LLMs succeed in approxing human judgments in a simple ToM paradigm, they fail at a logically equivalent task.
  arXiv  Detail & Related papers  (2026-02-12T16:33:58Z)
• RecToM: A Benchmark for Evaluating Machine Theory of Mind in LLM-based Conversational Recommender Systems [23.229692182223157]
  We propose RecToM, a novel benchmark for evaluating Large Language models.<n> RecToM focuses on two complementary dimensions: Cognitive Inference and Behavioral Prediction.<n>Extensive experiments on state-of-the-art LLMs demonstrate that RecToM poses a significant challenge.
  arXiv  Detail & Related papers  (2025-11-27T09:58:29Z)
• Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models? [14.29992535286614]
  Theory of Mind (ToM) is the ability to attribute mental states to others.<n>Recent advancements in Large Language Models have shown promising performance on ToM benchmarks.<n>Do these benchmarks necessitate explicit human-like reasoning processes, or can models succeed through alternative strategies?
  arXiv  Detail & Related papers  (2025-04-02T12:58:42Z)
• Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning [88.68573198200698]
  We introduce ExploreToM, the first framework to allow large-scale generation of diverse and challenging theory of mind data.<n>Our approach leverages an A* search over a custom domain-specific language to produce complex story structures and novel, diverse, yet plausible scenarios.<n>Our evaluation reveals that state-of-the-art LLMs, such as Llama-3.1-70B and GPT-4o, show accuracies as low as 0% and 9% on ExploreToM-generated data.
  arXiv  Detail & Related papers  (2024-12-12T21:29:00Z)
• NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding [55.38254464415964]
  Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states.
  arXiv  Detail & Related papers  (2024-04-21T11:51:13Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions [94.61530480991627]
  Theory of mind evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering.
  arXiv  Detail & Related papers  (2023-10-24T00:24:11Z)
• Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker [72.09076317574238]
  ToM is a plug-and-play approach to investigate the belief states of characters in reading comprehension. We show that ToM enhances off-the-shelf neural network theory mind in a zero-order setting while showing robust out-of-distribution performance compared to supervised baselines.
  arXiv  Detail & Related papers  (2023-06-01T17:24:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.