LLMs as Models for Analogical Reasoning

• URL: http://arxiv.org/abs/2406.13803v2
• Date: Tue, 18 Mar 2025 17:49:06 GMT
• Title: LLMs as Models for Analogical Reasoning
• Authors: Sam Musker, Alex Duchnowski, Raphaël Millière, Ellie Pavlick,
• Abstract summary: Analogical reasoning is fundamental to human cognition and learning.<n>Recent studies have shown that large language models can sometimes match humans in analogical reasoning tasks.
• Score: 14.412456982731467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Analogical reasoning-the capacity to identify and map structural relationships between different domains-is fundamental to human cognition and learning. Recent studies have shown that large language models (LLMs) can sometimes match humans in analogical reasoning tasks, opening the possibility that analogical reasoning might emerge from domain general processes. However, it is still debated whether these emergent capacities are largely superficial and limited to simple relations seen during training or whether they rather encompass the flexible representational and mapping capabilities which are the focus of leading cognitive models of analogy. In this study, we introduce novel analogical reasoning tasks that require participants to map between semantically contentful words and sequences of letters and other abstract characters. This task necessitates the ability to flexibly re-represent rich semantic information-an ability which is known to be central to human analogy but which is thus far not well-captured by existing cognitive theories and models. We assess the performance of both human participants and LLMs on tasks focusing on reasoning from semantic structure and semantic content, introducing variations that test the robustness of their analogical inferences. Advanced LLMs match human performance across several conditions, though humans and LLMs respond differently to certain task variations and semantic distractors. Our results thus provide new evidence that LLMs might offer a how-possibly explanation of human analogical reasoning in contexts that are not yet well modeled by existing theories, but that even today's best models are unlikely to yield how-actually explanations.

Related papers

• Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning [58.86928947970342]
  Embodied-R is a framework combining large-scale Vision-Language Models for perception and small-scale Language Models for reasoning. After training on only 5k embodied video samples, Embodied-R with a 3B LM matches state-of-the-art multimodal reasoning models. Embodied-R also exhibits emergent thinking patterns such as systematic analysis and contextual integration.
  arXiv  Detail & Related papers  (2025-04-17T06:16:11Z)
• LogiDynamics: Unraveling the Dynamics of Logical Inference in Large Language Model Reasoning [49.58786377307728]
  This paper adopts an exploratory approach by introducing a controlled evaluation environment for analogical reasoning. We analyze the comparative dynamics of inductive, abductive, and deductive inference pipelines. We investigate advanced paradigms such as hypothesis selection, verification, and refinement, revealing their potential to scale up logical inference.
  arXiv  Detail & Related papers  (2025-02-16T15:54:53Z)
• Thinking beyond the anthropomorphic paradigm benefits LLM research [1.7392902719515677]
  We analyze hundreds of thousands of computer science research articles from the past decade. We present empirical evidence of the prevalence and growth of anthropomorphic terminology in research on large language models (LLMs) We argue these conceptualizations may be limiting, and that challenging them opens up new pathways for understanding and improving LLMs beyond human analogies.
  arXiv  Detail & Related papers  (2025-02-13T11:32:09Z)
• Human-like conceptual representations emerge from language prediction [72.5875173689788]
  Large language models (LLMs) trained exclusively through next-token prediction over language data exhibit remarkably human-like behaviors. Are these models developing concepts akin to humans, and if so, how are such concepts represented and organized? Our results demonstrate that LLMs can flexibly derive concepts from linguistic descriptions in relation to contextual cues about other concepts. These findings establish that structured, human-like conceptual representations can naturally emerge from language prediction without real-world grounding.
  arXiv  Detail & Related papers  (2025-01-21T23:54:17Z)
• Large Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning [49.60849499134362]
  This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning) Traditional psycholinguistic evaluations often reflect statistical biases that may misrepresent LLMs' true linguistic capabilities. We introduce a neurolinguistic approach, utilizing a novel method that combines minimal pair and diagnostic probing to analyze activation patterns across model layers.
  arXiv  Detail & Related papers  (2024-11-12T04:16:44Z)
• Large Models of What? Mistaking Engineering Achievements for Human Linguistic Agency [0.11510009152620666]
  We argue that claims regarding linguistic capabilities of Large Language Models (LLMs) are based on at least two unfounded assumptions. Language completeness assumes that a distinct and complete thing such as a natural language' exists. The assumption of data completeness relies on the belief that a language can be quantified and wholly captured by data.
  arXiv  Detail & Related papers  (2024-07-11T18:06:01Z)
• Analyzing the Role of Semantic Representations in the Era of Large Language Models [104.18157036880287]
  We investigate the role of semantic representations in the era of large language models (LLMs) We propose an AMR-driven chain-of-thought prompting method, which we call AMRCoT. We find that it is difficult to predict which input examples AMR may help or hurt on, but errors tend to arise with multi-word expressions.
  arXiv  Detail & Related papers  (2024-05-02T17:32:59Z)
• PhonologyBench: Evaluating Phonological Skills of Large Language Models [57.80997670335227]
  Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. We present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs. We observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans.
  arXiv  Detail & Related papers  (2024-04-03T04:53:14Z)
• LLM-driven Imitation of Subrational Behavior : Illusion or Reality? [3.2365468114603937]
  Existing work highlights the ability of Large Language Models to address complex reasoning tasks and mimic human communication. We propose to investigate the use of LLMs to generate synthetic human demonstrations, which are then used to learn subrational agent policies. We experimentally evaluate the ability of our framework to model sub-rationality through four simple scenarios.
  arXiv  Detail & Related papers  (2024-02-13T19:46:39Z)
• From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoning [66.98861219674039]
  Heuristic-Analytic Reasoning (HAR) strategies drastically improve the coherence of rationalizations for model decisions. Our findings suggest that human-like reasoning strategies can effectively improve the coherence and reliability of PLM reasoning.
  arXiv  Detail & Related papers  (2023-10-24T19:46:04Z)
• Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Models [56.34029644009297]
  Large language models (LLMs) have demonstrated the ability to overcome various limitations of formal Knowledge Representation (KR) systems. LLMs excel most in abductive reasoning, followed by deductive reasoning, while they are least effective at inductive reasoning. We study single-task training, multi-task training, and "chain-of-thought" knowledge distillation fine-tuning technique to assess the performance of model.
  arXiv  Detail & Related papers  (2023-10-02T01:00:50Z)
• Agentivit\`a e telicit\`a in GilBERTo: implicazioni cognitive [77.71680953280436]
  The goal of this study is to investigate whether a Transformer-based neural language model infers lexical semantics. The semantic properties considered are telicity (also combined with definiteness) and agentivity.
  arXiv  Detail & Related papers  (2023-07-06T10:52:22Z)
• In-Context Analogical Reasoning with Pre-Trained Language Models [10.344428417489237]
  We explore the use of intuitive language-based abstractions to support analogy in AI systems. Specifically, we apply large pre-trained language models (PLMs) to visual Raven's Progressive Matrices ( RPM) We find that PLMs exhibit a striking capacity for zero-shot relational reasoning, exceeding human performance and nearing supervised vision-based methods.
  arXiv  Detail & Related papers  (2023-05-28T04:22:26Z)
• Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [75.85554779782048]
  Large Language Models (LLMs) have excited the natural language and machine learning community over recent years. Despite of numerous successful applications, the underlying mechanism of such in-context capabilities still remains unclear. In this work, we hypothesize that the learned textitsemantics of language tokens do the most heavy lifting during the reasoning process.
  arXiv  Detail & Related papers  (2023-05-24T07:33:34Z)
• The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative [7.03497683558609]
  Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. We present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC) Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning.
  arXiv  Detail & Related papers  (2022-10-24T13:01:24Z)
• Context Limitations Make Neural Language Models More Human-Like [32.488137777336036]
  We show discrepancies in context access between modern neural language models (LMs) and humans in incremental sentence processing. Additional context limitation was needed to make LMs better simulate human reading behavior. Our analyses also showed that human-LM gaps in memory access are associated with specific syntactic constructions.
  arXiv  Detail & Related papers  (2022-05-23T17:01:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.