Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground

• URL: http://arxiv.org/abs/2403.02451v2
• Date: Thu, 6 Jun 2024 00:30:01 GMT
• Title: Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground
• Authors: Adil Soubki, John Murzaku, Arash Yousefi Jordehi, Peter Zeng, Magdalena Markowska, Seyed Abolghasem Mirroshandel, Owen Rambow,
• Abstract summary: We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM.
• Score: 6.868969074841911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM.

Related papers

• Perceptions to Beliefs: Exploring Precursory Inferences for Theory of Mind in Large Language Models [51.91448005607405]
  We evaluate key human ToM precursors by annotating characters' perceptions on ToMi and FANToM. We present PercepToM, a novel ToM method leveraging LLMs' strong perception inference capability while supplementing their limited perception-to-belief inference.
  arXiv  Detail & Related papers  (2024-07-08T14:58:29Z)
• ToM-LM: Delegating Theory of Mind Reasoning to External Symbolic Executors in Large Language Models [5.455744338342196]
  Theory of Mind (ToM) refers to the ability of individuals to attribute mental states to others. Large Language Models (LLMs) have shown some promise with ToM ability, but they still struggle with complex ToM reasoning.
  arXiv  Detail & Related papers  (2024-04-23T20:59:03Z)
• NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding [55.38254464415964]
  Theory of mind evaluations currently focuses on testing models using machine-generated data or game settings prone to shortcuts and spurious correlations. We introduce NegotiationToM, a new benchmark designed to stress-test machine ToM in real-world negotiation surrounding covered multi-dimensional mental states.
  arXiv  Detail & Related papers  (2024-04-21T11:51:13Z)
• Look Before You Decide: Prompting Active Deduction of MLLMs for Assumptive Reasoning [68.83624133567213]
  We show that most prevalent MLLMs can be easily fooled by the introduction of a presupposition into the question. We also propose a simple yet effective method, Active Deduction (AD), to encourage the model to actively perform composite deduction.
  arXiv  Detail & Related papers  (2024-04-19T15:53:27Z)
• MMToM-QA: Multimodal Theory of Mind Question Answering [80.87550820953236]
  Theory of Mind (ToM) is an essential ingredient for developing machines with human-level social intelligence. Recent machine learning models, particularly large language models, seem to show some aspects of ToM understanding. Human ToM, on the other hand, is more than video or text understanding. People can flexibly reason about another person's mind based on conceptual representations extracted from any available data.
  arXiv  Detail & Related papers  (2024-01-16T18:59:24Z)
• InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models [50.03163753638256]
  Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark.
  arXiv  Detail & Related papers  (2023-11-20T07:06:31Z)
• Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Tests [1.099532646524593]
  We test 11 base- and instruction-tuned Large Language Models (LLMs) on capabilities relevant to Theory of Mind (ToM) We find that instruction-tuned LLMs from the GPT family outperform other models, and often also children. We suggest that the interlinked evolution and development of language and ToM may help explain what instruction-tuning adds.
  arXiv  Detail & Related papers  (2023-10-31T09:55:07Z)
• FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions [94.61530480991627]
  Theory of mind evaluations currently focus on testing models using passive narratives that inherently lack interactivity. We introduce FANToM, a new benchmark designed to stress-test ToM within information-asymmetric conversational contexts via question answering.
  arXiv  Detail & Related papers  (2023-10-24T00:24:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.