A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models

• URL: http://arxiv.org/abs/2503.05613v2
• Date: Fri, 06 Jun 2025 03:26:10 GMT
• Title: A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
• Authors: Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, Mengnan Du,
• Abstract summary: Large Language Models (LLMs) have transformed natural language processing, yet their internal mechanisms remain largely opaque.<n> mechanistic interpretability has attracted significant attention from the research community as a means to understand the inner workings of LLMs.<n>Sparse Autoencoders (SAEs) have emerged as a promising method due to their ability to disentangle the complex, superimposed features within LLMs into more interpretable components.
• Score: 40.67240575271987
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have transformed natural language processing, yet their internal mechanisms remain largely opaque. Recently, mechanistic interpretability has attracted significant attention from the research community as a means to understand the inner workings of LLMs. Among various mechanistic interpretability approaches, Sparse Autoencoders (SAEs) have emerged as a promising method due to their ability to disentangle the complex, superimposed features within LLMs into more interpretable components. This paper presents a comprehensive survey of SAEs for interpreting and understanding the internal workings of LLMs. Our major contributions include: (1) exploring the technical framework of SAEs, covering basic architecture, design improvements, and effective training strategies; (2) examining different approaches to explaining SAE features, categorized into input-based and output-based explanation methods; (3) discussing evaluation methods for assessing SAE performance, covering both structural and functional metrics; and (4) investigating real-world applications of SAEs in understanding and manipulating LLM behaviors.

Related papers

• Large Language Models for History, Philosophy, and Sociology of Science: Interpretive Uses, Methodological Challenges, and Critical Perspectives [0.0]
  This paper explores the use of large language models (LLMs) as research tools in the history, philosophy, and sociology of science (HPSS)<n>LLMs are remarkably effective at processing unstructured text and inferring meaning from context.<n>This raises both opportunities and challenges for HPSS, which emphasizes interpretive methodologies and understands meaning as context-dependent, ambiguous, and historically situated.
  arXiv  Detail & Related papers  (2025-06-13T21:44:13Z)
• Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Models [45.05285463251872]
  We introduce a novel learning paradigm -- Modular Machine Learning (MML) -- as an essential approach toward new-generation large language models (LLMs) MML decomposes the complex structure of LLMs into three interdependent components: modular representation, modular model, and modular reasoning. We present a feasible implementation of MML-based LLMs via leveraging advanced techniques such as disentangled representation learning, neural architecture search and neuro-symbolic learning.
  arXiv  Detail & Related papers  (2025-04-28T17:42:02Z)
• How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspective [64.00022624183781]
  Large language models (LLMs) can assess relevance and support information retrieval (IR) tasks. We investigate how different LLM modules contribute to relevance judgment through the lens of mechanistic interpretability.
  arXiv  Detail & Related papers  (2025-04-10T16:14:55Z)
• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models [50.587868616659826]
  Sparse Autoencoders (SAEs) have been shown to enhance interpretability and steerability in Large Language Models (LLMs) In this work, we extend the application of SAEs to Vision-Language Models (VLMs), such as CLIP, and introduce a comprehensive framework for evaluating monosemanticity in vision representations.
  arXiv  Detail & Related papers  (2025-04-03T17:58:35Z)
• How LLMs Learn: Tracing Internal Representations with Sparse Autoencoders [30.36521888592164]
  Large Language Models (LLMs) demonstrate remarkable multilingual capabilities and broad knowledge. We analyze how the information encoded in LLMs' internal representations evolves during the training process.
  arXiv  Detail & Related papers  (2025-03-09T02:13:44Z)
• FAC$^2$E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition [56.76951887823882]
  Large language models (LLMs) are primarily evaluated by overall performance on various text understanding and generation tasks. We present FAC$2$E, a framework for Fine-grAined and Cognition-grounded LLMs' Capability Evaluation.
  arXiv  Detail & Related papers  (2024-02-29T21:05:37Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• Towards Uncovering How Large Language Model Works: An Explainability Perspective [38.07611356855978]
  Large language models (LLMs) have led to breakthroughs in language tasks, yet the internal mechanisms that enable their remarkable generalization and reasoning abilities remain opaque. This paper aims to uncover the mechanisms underlying LLM functionality through the lens of explainability.
  arXiv  Detail & Related papers  (2024-02-16T13:46:06Z)
• Large Language Model-Based Interpretable Machine Learning Control in Building Energy Systems [3.0309252269809264]
  This paper investigates and explores Interpretable Machine Learning (IML), a branch of Machine Learning (ML) that enhances transparency and understanding of models and their inferences. We develop an innovative framework that combines the principles of Shapley values and the in-context learning feature of Large Language Models (LLMs) The paper presents a case study to demonstrate the feasibility of the developed IML framework for model predictive control-based precooling under demand response events in a virtual testbed.
  arXiv  Detail & Related papers  (2024-02-14T21:19:33Z)
• Rethinking Interpretability in the Era of Large Language Models [76.1947554386879]
  Large language models (LLMs) have demonstrated remarkable capabilities across a wide array of tasks. The capability to explain in natural language allows LLMs to expand the scale and complexity of patterns that can be given to a human. These new capabilities raise new challenges, such as hallucinated explanations and immense computational costs.
  arXiv  Detail & Related papers  (2024-01-30T17:38:54Z)
• From Understanding to Utilization: A Survey on Explainability for Large Language Models [27.295767173801426]
  This survey underscores the imperative for increased explainability in Large Language Models (LLMs) Our focus is primarily on pre-trained Transformer-based LLMs, which pose distinctive interpretability challenges due to their scale and complexity. When considering the utilization of explainability, we explore several compelling methods that concentrate on model editing, control generation, and model enhancement.
  arXiv  Detail & Related papers  (2024-01-23T16:09:53Z)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention [53.896974148579346]
  Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. The enigmatic black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. We propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs.
  arXiv  Detail & Related papers  (2023-12-22T19:55:58Z)
• Position Paper: Bridging the Gap Between Machine Learning and Sensitivity Analysis [9.191045750996526]
  We argue that interpretations of machine learning (ML) models can be seen as a form of sensitivity analysis (SA) We call attention to the benefits of a unified SA-based view of explanations in ML and the necessity to fully credit related work.
  arXiv  Detail & Related papers  (2023-12-20T17:59:11Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.