Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures

• URL: http://arxiv.org/abs/2604.16042v2
• Date: Mon, 20 Apr 2026 05:23:39 GMT
• Title: Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures
• Abstract summary: In intrinsic interpretability, which builds transparency directly into model architectures and computations, has emerged as a promising alternative.<n>This paper presents a systematic review of the recent advances in intrinsic interpretability for Large Language Models.
• Score: 37.785583281358875
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Large Language Models (LLMs) have achieved strong performance across many NLP tasks, their opaque internal mechanisms hinder trustworthiness and safe deployment. Existing surveys in explainable AI largely focus on post-hoc explanation methods that interpret trained models through external approximations. In contrast, intrinsic interpretability, which builds transparency directly into model architectures and computations, has recently emerged as a promising alternative. This paper presents a systematic review of the recent advances in intrinsic interpretability for LLMs, categorizing existing approaches into five design paradigms: functional transparency, concept alignment, representational decomposability, explicit modularization, and latent sparsity induction. We further discuss open challenges and outline future research directions in this emerging field. The paper list is available at: https://github.com/PKU-PILLAR-Group/Survey-Intrinsic-Interpretability-of-LLMs.

Related papers

• Applied Explainability for Large Language Models: A Comparative Study [0.0]
  This paper presents an applied comparative study of three explainability techniques: Integrated Gradients, Attention Rollout, and SHAP.<n>The results show that gradient-based attribution provides more stable and intuitive explanations.<n>Work highlights key trade-offs between explainability methods and emphasizes their role as diagnostic tools rather than definitive explanations.
  arXiv  Detail & Related papers  (2026-04-15T13:07:29Z)
• Implicit Reasoning in Large Language Models: A Comprehensive Survey [67.53966514728383]
  Large Language Models (LLMs) have demonstrated strong generalization across a wide range of tasks.<n>Recent studies have shifted attention from explicit chain-of-thought prompting toward implicit reasoning.<n>This survey introduces a taxonomy centered on execution paradigms, shifting the focus from representational forms to computational strategies.
  arXiv  Detail & Related papers  (2025-09-02T14:16:02Z)
• A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models [50.34089812436633]
  Large Language Models (LLMs) have transformed natural language processing, yet their internal mechanisms remain largely opaque.<n> mechanistic interpretability has attracted significant attention from the research community as a means to understand the inner workings of LLMs.<n>Sparse Autoencoders (SAEs) have emerged as a promising method due to their ability to disentangle the complex, superimposed features within LLMs into more interpretable components.
  arXiv  Detail & Related papers  (2025-03-07T17:38:00Z)
• A Survey of Model Architectures in Information Retrieval [59.61734783818073]
  The period from 2019 to the present has represented one of the biggest paradigm shifts in information retrieval (IR) and natural language processing (NLP)<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude with a forward-looking discussion of emerging challenges and future directions.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• Retrieval-Augmented Semantic Parsing: Improving Generalization with Lexical Knowledge [6.948555996661213]
  We introduce Retrieval-Augmented Semantic Parsing (RASP), a simple yet effective approach that integrates external symbolic knowledge into the parsing process.<n>Our experiments show that LLMs outperform previous encoder-decoder baselines for semantic parsing.<n>RASP further enhances their ability to predict unseen concepts, nearly doubling the performance of previous models on out-of-distribution concepts.
  arXiv  Detail & Related papers  (2024-12-13T15:30:20Z)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention [53.896974148579346]
  Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. The enigmatic black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. We propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs.
  arXiv  Detail & Related papers  (2023-12-22T19:55:58Z)
• Proto-lm: A Prototypical Network-Based Framework for Built-in Interpretability in Large Language Models [27.841725567976315]
  Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), but their lack of interpretability has been a major concern. In this work, we introduce proto-lm, a prototypical network-based white-box framework that allows LLMs to learn immediately interpretable embeddings. Our method's applicability and interpretability are demonstrated through experiments on a wide range of NLP tasks, and our results indicate a new possibility of creating interpretable models without sacrificing performance.
  arXiv  Detail & Related papers  (2023-11-03T05:55:32Z)
• Stable and Interpretable Deep Learning for Tabular Data: Introducing InterpreTabNet with the Novel InterpreStability Metric [4.362293468843233]
  We introduce InterpreTabNet, a model designed to enhance both classification accuracy and interpretability. We also present a novel evaluation metric, InterpreStability, which quantifies the stability of a model's interpretability.
  arXiv  Detail & Related papers  (2023-10-04T15:04:13Z)
• Explainability for Large Language Models: A Survey [59.67574757137078]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. This paper introduces a taxonomy of explainability techniques and provides a structured overview of methods for explaining Transformer-based language models.
  arXiv  Detail & Related papers  (2023-09-02T22:14:26Z)
• Convex Polytope Modelling for Unsupervised Derivation of Semantic Structure for Data-efficient Natural Language Understanding [31.888489552069146]
  A Convex-Polytopic-Model-based framework shows great potential in automatically extracting semantic patterns by exploiting the raw dialog corpus. We show that this framework can exploit semantic-frame-related features in the corpus, reveal the underlying semantic structure of the utterances, and boost the performance of the state-of-the-art NLU model with minimal supervision.
  arXiv  Detail & Related papers  (2022-01-25T19:12:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.