A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks

• URL: http://arxiv.org/abs/2502.08796v1
• Date: Wed, 12 Feb 2025 21:19:30 GMT
• Title: A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks
• Authors: Karahan Sarıtaş, Kıvanç Tezören, Yavuz Durmazkeser,
• Abstract summary: This systematic review synthesizes current efforts to assess large language models' (LLMs) ability to perform ToM tasks. A recurring theme in the literature reveals that while LLMs demonstrate emerging competence in ToM tasks, significant gaps persist in their emulation of human cognitive abilities.
• Score: 0.0
• License:
• Abstract: In recent years, evaluating the Theory of Mind (ToM) capabilities of large language models (LLMs) has received significant attention within the research community. As the field rapidly evolves, navigating the diverse approaches and methodologies has become increasingly complex. This systematic review synthesizes current efforts to assess LLMs' ability to perform ToM tasks, an essential aspect of human cognition involving the attribution of mental states to oneself and others. Despite notable advancements, the proficiency of LLMs in ToM remains a contentious issue. By categorizing benchmarks and tasks through a taxonomy rooted in cognitive science, this review critically examines evaluation techniques, prompting strategies, and the inherent limitations of LLMs in replicating human-like mental state reasoning. A recurring theme in the literature reveals that while LLMs demonstrate emerging competence in ToM tasks, significant gaps persist in their emulation of human cognitive abilities.

Related papers

• Humanlike Cognitive Patterns as Emergent Phenomena in Large Language Models [2.9312156642007294]
  We systematically review Large Language Models' capabilities across three important cognitive domains: decision-making biases, reasoning, and creativity. On decision-making, our synthesis reveals that while LLMs demonstrate several human-like biases, some biases observed in humans are absent. On reasoning, advanced LLMs like GPT-4 exhibit deliberative reasoning akin to human System-2 thinking, while smaller models fall short of human-level performance. A distinct dichotomy emerges in creativity: while LLMs excel in language-based creative tasks, such as storytelling, they struggle with divergent thinking tasks that require real-world context.
  arXiv  Detail & Related papers  (2024-12-20T02:26:56Z)
• A Survey on Human-Centric LLMs [11.49752599240738]
  Large language models (LLMs) can simulate human cognition and behavior. This survey focuses on their performance in both individual tasks and collective tasks.
  arXiv  Detail & Related papers  (2024-11-20T12:34:44Z)
• Understanding the Role of LLMs in Multimodal Evaluation Benchmarks [77.59035801244278]
  This paper investigates the role of the Large Language Model (LLM) backbone in Multimodal Large Language Models (MLLMs) evaluation. Our study encompasses four diverse MLLM benchmarks and eight state-of-the-art MLLMs. Key findings reveal that some benchmarks allow high performance even without visual inputs and up to 50% of error rates can be attributed to insufficient world knowledge in the LLM backbone.
  arXiv  Detail & Related papers  (2024-10-16T07:49:13Z)
• Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges [14.739357670600102]
  This comprehensive review explores the intersection of Large Language Models (LLMs) and cognitive science. We analyze methods for evaluating LLMs cognitive abilities and discuss their potential as cognitive models. We assess cognitive biases and limitations of LLMs, along with proposed methods for improving their performance.
  arXiv  Detail & Related papers  (2024-09-04T02:30:12Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making. We present a process-based benchmark MR-Ben that demands a meta-reasoning skill. Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• ToMBench: Benchmarking Theory of Mind in Large Language Models [41.565202027904476]
  ToM is the cognitive capability to perceive and ascribe mental states to oneself and others. Existing ToM evaluations are hindered by challenges such as constrained scope, subjective judgment, and unintended contamination. We introduce ToMBench with three key characteristics: a systematic evaluation framework encompassing 8 tasks and 31 abilities in social cognition, a multiple-choice question format to support automated and unbiased evaluation, and a build-from-scratch bilingual inventory to strictly avoid data leakage.
  arXiv  Detail & Related papers  (2024-02-23T02:05:46Z)
• Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities [63.90227161974381]
  SimToM is a novel prompting framework inspired by Simulation Theory's notion of perspective-taking. Our approach, which requires no additional training and minimal prompt-tuning, shows substantial improvement over existing methods.
  arXiv  Detail & Related papers  (2023-11-16T22:49:27Z)
• Exploring the Cognitive Knowledge Structure of Large Language Models: An Educational Diagnostic Assessment Approach [50.125704610228254]
  Large Language Models (LLMs) have not only exhibited exceptional performance across various tasks, but also demonstrated sparks of intelligence. Recent studies have focused on assessing their capabilities on human exams and revealed their impressive competence in different domains. We conduct an evaluation using MoocRadar, a meticulously annotated human test dataset based on Bloom taxonomy.
  arXiv  Detail & Related papers  (2023-10-12T09:55:45Z)
• A Survey on Evaluation of Large Language Models [87.60417393701331]
  Large language models (LLMs) are gaining increasing popularity in both academia and industry. This paper focuses on three key dimensions: what to evaluate, where to evaluate, and how to evaluate.
  arXiv  Detail & Related papers  (2023-07-06T16:28:35Z)
• Understanding Social Reasoning in Language Models with Language Models [34.068368860882586]
  We present a novel framework for generating evaluations with Large Language Models (LLMs) by populating causal templates. We create a new social reasoning benchmark (BigToM) for LLMs which consists of 25 controls and 5,000 model-written evaluations. We find that human participants rate the quality of our benchmark higher than previous crowd-sourced evaluations and comparable to expert-written evaluations.
  arXiv  Detail & Related papers  (2023-06-21T16:42:15Z)
• Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models [82.50173296858377]
  Many anecdotal examples were used to suggest newer large language models (LLMs) like ChatGPT and GPT-4 exhibit Neural Theory-of-Mind (N-ToM) We investigate the extent of LLMs' N-ToM through an extensive evaluation on 6 tasks and find that while LLMs exhibit certain N-ToM abilities, this behavior is far from being robust.
  arXiv  Detail & Related papers  (2023-05-24T06:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.