Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models

• URL: http://arxiv.org/abs/2310.19619v2
• Date: Thu, 26 Dec 2024 20:04:21 GMT
• Title: Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models
• Authors: Ziqiao Ma, Jacob Sansom, Run Peng, Joyce Chai,
• Abstract summary: Large Language Models (LLMs) have generated considerable interest and debate regarding their potential emergence of Theory of Mind (ToM) Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand to develop new benchmarks. We taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM.
• Score: 13.265149793053546
• License:
• Abstract: Large Language Models (LLMs) have generated considerable interest and debate regarding their potential emergence of Theory of Mind (ToM). Several recent inquiries reveal a lack of robust ToM in these models and pose a pressing demand to develop new benchmarks, as current ones primarily focus on different aspects of ToM and are prone to shortcuts and data leakage. In this position paper, we seek to answer two road-blocking questions: (1) How can we taxonomize a holistic landscape of machine ToM? (2) What is a more effective evaluation protocol for machine ToM? Following psychological studies, we taxonomize machine ToM into 7 mental state categories and delineate existing benchmarks to identify under-explored aspects of ToM. We argue for a holistic and situated evaluation of ToM to break ToM into individual components and treat LLMs as an agent who is physically situated in environments and socially situated in interactions with humans. Such situated evaluation provides a more comprehensive assessment of mental states and potentially mitigates the risk of shortcuts and data leakage. We further present a pilot study in a grid world setup as a proof of concept. We hope this position paper can facilitate future research to integrate ToM with LLMs and offer an intuitive means for researchers to better position their work in the landscape of ToM. Project page: https://github.com/Mars-tin/awesome-theory-of-mind

Related papers

• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency [63.23935582919081]
  Chain-of-Thought (CoT) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs) We introduce MME-CoT, a specialized benchmark evaluating the CoT reasoning performance of LMMs. We conduct an in-depth analysis of state-of-the-art LMMs, uncovering several key insights.
  arXiv  Detail & Related papers  (2025-02-13T18:59:46Z)
• The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characters [67.61587661660852]
  Theory-of-Mind (ToM) allows humans to understand and interpret the mental states of others. In this paper, we verify the importance of understanding long personal backgrounds in ToM. We assess the performance of machines' ToM capabilities in realistic evaluation scenarios.
  arXiv  Detail & Related papers  (2025-01-03T09:04:45Z)
• Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning [13.545981051703682]
  Theory of Mind (ToM) capabilities in LLMs have recently become a central object of investigation. We identify several lines of work in different communities in AI, including LLM benchmarking, ToM add-ons, ToM probing, and formal models for ToM. We conclude with suggestions for improved evaluation of ToM capabilities inspired by dynamic environments used in cognitive tasks.
  arXiv  Detail & Related papers  (2024-12-18T09:06:48Z)
• NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding [55.38254464415964]
  Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states.
  arXiv  Detail & Related papers  (2024-04-21T11:51:13Z)
• MMToM-QA: Multimodal Theory of Mind Question Answering [80.87550820953236]
  Theory of Mind (ToM) is an essential ingredient for developing machines with human-level social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person's mind based on conceptual representations extracted from any available data.
  arXiv  Detail & Related papers  (2024-01-16T18:59:24Z)
• Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilities [63.90227161974381]
  SimToM is a novel prompting framework inspired by Simulation Theory's notion of perspective-taking. Our approach, which requires no additional training and minimal prompt-tuning, shows substantial improvement over existing methods.
  arXiv  Detail & Related papers  (2023-11-16T22:49:27Z)
• FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions [94.61530480991627]
  Theory of mind evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering.
  arXiv  Detail & Related papers  (2023-10-24T00:24:11Z)
• A Review on Machine Theory of Mind [16.967933605635203]
  Theory of Mind (ToM) is the ability to attribute mental states to others, the basis of human cognition. In this paper, we review recent progress in machine ToM on beliefs, desires, and intentions.
  arXiv  Detail & Related papers  (2023-03-21T04:58:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.