Mechanistic Indicators of Understanding in Large Language Models

• URL: http://arxiv.org/abs/2507.08017v3
• Date: Thu, 24 Jul 2025 12:23:53 GMT
• Title: Mechanistic Indicators of Understanding in Large Language Models
• Authors: Pierre Beckmann, Matthieu Queloz,
• Abstract summary: We argue that Large Language Models (LLMs) develop internal structures that are functionally analogous to the kind of understanding that consists in seeing connections.<n> conceptual understanding emerges when a model forms "features" as directions in latent space, learning the connections between diverse manifestations of something.<n>Second, state-of-the-world understanding emerges when a model learns contingent factual connections between features and dynamically tracks changes in the world.<n>Third, principled understanding emerges when a model ceases to rely on a collection of memorized facts and discovers a "circuit" connecting these facts.
• Score: 2.752171077382186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent findings in mechanistic interpretability (MI), the field probing the inner workings of Large Language Models (LLMs), challenge the view that these models rely solely on superficial statistics. We offer an accessible synthesis of these findings that doubles as an introduction to MI while integrating these findings within a novel theoretical framework for thinking about machine understanding. We argue that LLMs develop internal structures that are functionally analogous to the kind of understanding that consists in seeing connections. To sharpen this idea, we propose a three-tiered conception of understanding. First, conceptual understanding emerges when a model forms "features" as directions in latent space, learning the connections between diverse manifestations of something. Second, state-of-the-world understanding emerges when a model learns contingent factual connections between features and dynamically tracks changes in the world. Third, principled understanding emerges when a model ceases to rely on a collection of memorized facts and discovers a "circuit" connecting these facts. However, these forms of understanding remain radically different from human understanding, as the phenomenon of "parallel mechanisms" shows. We conclude that the debate should move beyond the yes-or-no question of whether LLMs understand to investigate how their strange minds work and forge conceptions that fit them.

Related papers

• Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging [32.70038648928894]
  Vision-Language Models (VLMs) combine visual perception with the general capabilities, such as reasoning, of Large Language Models (LLMs)<n>In this work, we explore to compose perception and reasoning through model merging that connects parameters of different models.<n>We find that perception capabilities are predominantly encoded in the early layers of the model, whereas reasoning is largely facilitated by the middle-to-late layers.
  arXiv  Detail & Related papers  (2025-05-08T17:56:23Z)
• Beyond Pattern Recognition: Probing Mental Representations of LMs [9.461066161954077]
  Language Models (LMs) have demonstrated impressive capabilities in solving complex reasoning tasks.<n>We propose to delve deeper into the mental model of various LMs.
  arXiv  Detail & Related papers  (2025-02-23T21:20:28Z)
• Failure Modes of LLMs for Causal Reasoning on Narratives [51.19592551510628]
  We investigate the interaction between world knowledge and logical reasoning.<n>We find that state-of-the-art large language models (LLMs) often rely on superficial generalizations.<n>We show that simple reformulations of the task can elicit more robust reasoning behavior.
  arXiv  Detail & Related papers  (2024-10-31T12:48:58Z)
• Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference [13.59675117792588]
  Recent studies on language models (LMs) have sparked a debate on whether they can learn systematic inferential principles.<n>This paper presents a mechanistic interpretation of syllogistic inference.
  arXiv  Detail & Related papers  (2024-08-16T07:47:39Z)
• The Cognitive Revolution in Interpretability: From Explaining Behavior to Interpreting Representations and Algorithms [3.3653074379567096]
  mechanistic interpretability (MI) has emerged as a distinct research area studying the features and implicit algorithms learned by foundation models such as large language models. We argue that current methods are ripe to facilitate a transition in deep learning interpretation echoing the "cognitive revolution" in 20th-century psychology. We propose a taxonomy mirroring key parallels in computational neuroscience to describe two broad categories of MI research.
  arXiv  Detail & Related papers  (2024-08-11T20:50:16Z)
• Aligned at the Start: Conceptual Groupings in LLM Embeddings [10.282327560070202]
  This paper shifts focus to the often-overlooked input embeddings - the initial representations fed into transformer blocks.<n>Using fuzzy graph, k-nearest neighbor (k-NN), and community detection, we analyze embeddings from diverse LLMs.
  arXiv  Detail & Related papers  (2024-06-08T01:27:19Z)
• What does the Knowledge Neuron Thesis Have to do with Knowledge? [13.651280182588666]
  We reassess the Knowledge Neuron (KN): an interpretation of the mechanism underlying the ability of large language models to recall facts from a training corpus. We find that this thesis is, at best, an oversimplification.
  arXiv  Detail & Related papers  (2024-05-03T18:34:37Z)
• Identifying Semantic Induction Heads to Understand In-Context Learning [103.00463655766066]
  We investigate whether attention heads encode two types of relationships between tokens present in natural languages. We find that certain attention heads exhibit a pattern where, when attending to head tokens, they recall tail tokens and increase the output logits of those tail tokens.
  arXiv  Detail & Related papers  (2024-02-20T14:43:39Z)
• Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals [82.68757839524677]
  Interpretability research aims to bridge the gap between empirical success and our scientific understanding of large language models (LLMs) We propose a formulation of competition of mechanisms, which focuses on the interplay of multiple mechanisms instead of individual mechanisms. Our findings show traces of the mechanisms and their competition across various model components and reveal attention positions that effectively control the strength of certain mechanisms.
  arXiv  Detail & Related papers  (2024-02-18T17:26:51Z)
• Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models [80.32412260877628]
  We study how to learn human-interpretable concepts from data.<n> Weaving together ideas from both fields, we show that concepts can be provably recovered from diverse data.
  arXiv  Detail & Related papers  (2024-02-14T15:23:59Z)
• Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation. We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
  arXiv  Detail & Related papers  (2023-11-08T20:41:18Z)
• Explainability for Large Language Models: A Survey [59.67574757137078]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. This paper introduces a taxonomy of explainability techniques and provides a structured overview of methods for explaining Transformer-based language models.
  arXiv  Detail & Related papers  (2023-09-02T22:14:26Z)
• Foundational Models Defining a New Era in Vision: A Survey and Outlook [151.49434496615427]
  Vision systems to see and reason about the compositional nature of visual scenes are fundamental to understanding our world. The models learned to bridge the gap between such modalities coupled with large-scale training data facilitate contextual reasoning, generalization, and prompt capabilities at test time. The output of such models can be modified through human-provided prompts without retraining, e.g., segmenting a particular object by providing a bounding box, having interactive dialogues by asking questions about an image or video scene or manipulating the robot's behavior through language instructions.
  arXiv  Detail & Related papers  (2023-07-25T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.