How Do LLMs Perform Two-Hop Reasoning in Context?

• URL: http://arxiv.org/abs/2502.13913v1
• Date: Wed, 19 Feb 2025 17:46:30 GMT
• Title: How Do LLMs Perform Two-Hop Reasoning in Context?
• Authors: Tianyu Guo, Hanlin Zhu, Ruiqi Zhang, Jiantao Jiao, Song Mei, Michael I. Jordan, Stuart Russell,
• Abstract summary: We train a three-layer transformer on synthetic two-hop reasoning tasks. We explain the inner mechanisms for how models learn to randomly guess between distractions. Our findings provide new insights into how reasoning emerges during training.
• Score: 76.79936191530784
• License:
• Abstract: "Socrates is human. All humans are mortal. Therefore, Socrates is mortal." This classical example demonstrates two-hop reasoning, where a conclusion logically follows from two connected premises. While transformer-based Large Language Models (LLMs) can make two-hop reasoning, they tend to collapse to random guessing when faced with distracting premises. To understand the underlying mechanism, we train a three-layer transformer on synthetic two-hop reasoning tasks. The training dynamics show two stages: a slow learning phase, where the 3-layer transformer performs random guessing like LLMs, followed by an abrupt phase transitions, where the 3-layer transformer suddenly reaches $100%$ accuracy. Through reverse engineering, we explain the inner mechanisms for how models learn to randomly guess between distractions initially, and how they learn to ignore distractions eventually. We further propose a three-parameter model that supports the causal claims for the mechanisms to the training dynamics of the transformer. Finally, experiments on LLMs suggest that the discovered mechanisms generalize across scales. Our methodologies provide new perspectives for scientific understandings of LLMs and our findings provide new insights into how reasoning emerges during training.

Related papers

• Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment [54.62926010621013]
  We introduce a novel task, code reasoning, to provide a new perspective for the reasoning abilities of large language models. We summarize three meta-benchmarks based on established forms of logical reasoning, and instantiate these into eight specific benchmark tasks. We present a new pathway exploration pipeline inspired by human intricate problem-solving methods.
  arXiv  Detail & Related papers  (2025-02-17T10:39:58Z)
• How Transformers Solve Propositional Logic Problems: A Mechanistic Analysis [16.65073455206535]
  Large language models (LLMs) have shown amazing performance on tasks that require planning and reasoning. Motivated by this, we investigate the internal mechanisms that underpin a network's ability to perform complex logical reasoning.
  arXiv  Detail & Related papers  (2024-11-06T18:35:32Z)
• On Memorization of Large Language Models in Logical Reasoning [70.94164038947078]
  Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. One hypothesis is that the increasingly high and nearly saturated performance could be due to the memorization of similar problems. We show that fine-tuning leads to heavy memorization, but it also consistently improves generalization performance.
  arXiv  Detail & Related papers  (2024-10-30T15:31:54Z)
• Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization [22.033370572209744]
  We study whether transformers can learn to implicitly reason over parametric knowledge. We focus on two representative reasoning types, composition and comparison. We find that transformers can learn implicit reasoning, but only through grokking.
  arXiv  Detail & Related papers  (2024-05-23T21:42:19Z)
• Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics [45.69328374321502]
  Auto-regressive large language models (LLMs) show impressive capacities to solve many complex reasoning tasks. LLMs fail to conclude '$B gets A$' during inference even if the two sentences are semantically identical. We theoretically analyze the reversal curse via the training dynamics of gradient descent for two auto-regressive models.
  arXiv  Detail & Related papers  (2024-05-07T21:03:51Z)
• How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning [44.02173413922695]
  A lack of understanding prevails around the internal mechanisms of the models that facilitate Chain-of-Thought (CoT) prompting. This work investigates the sub-structures within Large Language Models that manifest CoT reasoning from a point of view.
  arXiv  Detail & Related papers  (2024-02-28T13:14:20Z)
• Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregation [110.71955853831707]
  We view LMs as deriving new conclusions by aggregating indirect reasoning paths seen at pre-training time. We formalize the reasoning paths as random walk paths on the knowledge/reasoning graphs. Experiments and analysis on multiple KG and CoT datasets reveal the effect of training on random walk paths.
  arXiv  Detail & Related papers  (2024-02-05T18:25:51Z)
• Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination? [73.454943870226]
  This work studies a specific type of hallucination induced by semantic associations. To quantify this phenomenon, we propose a novel probing method and benchmark called EureQA.
  arXiv  Detail & Related papers  (2023-11-16T09:27:36Z)
• Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models [107.07851578154242]
  Language models (LMs) have strong multi-step (i.e., procedural) reasoning capabilities. It is unclear whether LMs perform tasks by cheating with answers memorized from pretraining corpus, or, via a multi-step reasoning mechanism. We show that MechanisticProbe is able to detect the information of the reasoning tree from the model's attentions for most examples.
  arXiv  Detail & Related papers  (2023-10-23T01:47:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.