Enhancing Automated Interpretability with Output-Centric Feature Descriptions

• URL: http://arxiv.org/abs/2501.08319v1
• Date: Tue, 14 Jan 2025 18:53:00 GMT
• Title: Enhancing Automated Interpretability with Output-Centric Feature Descriptions
• Abstract summary: We propose efficient, output-centric methods for automatically generating feature descriptions.<n>Our descriptions better capture the causal effect of a feature on model outputs than input-centric descriptions.<n>We show that output-centric descriptions can be used to find inputs that activate features previously thought to be "dead"
• Score: 14.591455690954513
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated interpretability pipelines generate natural language descriptions for the concepts represented by features in large language models (LLMs), such as plants or the first word in a sentence. These descriptions are derived using inputs that activate the feature, which may be a dimension or a direction in the model's representation space. However, identifying activating inputs is costly, and the mechanistic role of a feature in model behavior is determined both by how inputs cause a feature to activate and by how feature activation affects outputs. Using steering evaluations, we reveal that current pipelines provide descriptions that fail to capture the causal effect of the feature on outputs. To fix this, we propose efficient, output-centric methods for automatically generating feature descriptions. These methods use the tokens weighted higher after feature stimulation or the highest weight tokens after applying the vocabulary "unembedding" head directly to the feature. Our output-centric descriptions better capture the causal effect of a feature on model outputs than input-centric descriptions, but combining the two leads to the best performance on both input and output evaluations. Lastly, we show that output-centric descriptions can be used to find inputs that activate features previously thought to be "dead".

Related papers

• Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects [58.98080129016082]
  Feature-Effect Geometry Analysis (FEGA) is an unsupervised framework that removes the same active SAE feature across contexts.<n>We show that a feature can be interpretable and causally relevant without providing a stable direction for steering.
  arXiv  Detail & Related papers  (2026-07-27T16:45:08Z)
• ADAG: Automatically Describing Attribution Graphs [50.460651620833055]
  We introduce textbfADAG, an end-to-end pipeline for describing these attribution graphs which is fully automated.<n>We then introduce a novel clustering algorithm for grouping features, and an LLM explainer--simulator setup which generates and scores natural-language explanations of the functional role of these feature groups.<n>We run our system on known human-analysed circuit-tracing tasks and recover interpretable circuits, and further show ADAG can find steerable clusters which are responsible for a harmful advice jailbreak in Llama 3.1 8B Instruct.
  arXiv  Detail & Related papers  (2026-04-08T21:34:37Z)
• Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features [1.5874067490843806]
  Control Reinforcement Learning trains a policy to select SAE features for steering at each token, producing interpretable intervention logs.<n> Adaptive Feature Masking encourages diverse feature discovery while preserving singlefeature interpretability.<n>On Gemma 2 2B across MMLU, BBQ, GSM8K, HarmBench, and XSTest, CRL achieves improvements while providing per-token intervention logs.
  arXiv  Detail & Related papers  (2026-02-11T02:28:49Z)
• Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features [11.463277740376236]
  Current interpretation methods infer feature semantics from activation patterns, but overlook that features are trained to reconstruct activations that serve computational roles in the forward pass.<n>We introduce a novel weight-based interpretation framework that measures functional effects through direct weight interactions, requiring no activation data.
  arXiv  Detail & Related papers  (2026-01-30T01:30:48Z)
• Mitigating Multimodal Hallucinations via Gradient-based Self-Reflection [49.26064449816502]
  We propose a Gradient-based Influence-Aware Constrained Decoding (GACD) method to address text-visual bias and co-occurrence bias.<n>GACD effectively reduces hallucinations and improves the visual grounding of MLLM outputs.
  arXiv  Detail & Related papers  (2025-09-03T08:13:52Z)
• SAEs Are Good for Steering -- If You Select the Right Features [45.47261543304217]
  Current methods identify SAE features to steer by analyzing the input tokens that activate them.<n>In this work, we draw a distinction between two types of features: input features, which mainly capture patterns in the model's input, and output features, which have a human-understandable effect on the model's output.
  arXiv  Detail & Related papers  (2025-05-26T14:47:59Z)
• Features that Make a Difference: Leveraging Gradients for Improved Dictionary Learning [4.051777802443125]
  Sparse Autoencoders (SAEs) are a promising approach for extracting neural network representations. We introduce Gradient SAEs, which modify the $k$-sparse autoencoder architecture by augmenting the TopK activation function. We find evidence that g-SAEs learn latents that are on average more effective at steering models in arbitrary contexts.
  arXiv  Detail & Related papers  (2024-11-15T18:03:52Z)
• Adaptive Language-Guided Abstraction from Contrastive Explanations [53.48583372522492]
  It is necessary to determine which features of the environment are relevant before determining how these features should be used to compute reward. End-to-end methods for joint feature and reward learning often yield brittle reward functions that are sensitive to spurious state features. This paper describes a method named ALGAE which alternates between using language models to iteratively identify human-meaningful features.
  arXiv  Detail & Related papers  (2024-09-12T16:51:58Z)
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models [55.19497659895122]
  We introduce methods for discovering and applying sparse feature circuits. These are causally implicatedworks of human-interpretable features for explaining language model behaviors.
  arXiv  Detail & Related papers  (2024-03-28T17:56:07Z)
• ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models [9.261590575715669]
  We develop anAttribute-Guided Prompt Tuning (ArGue) for Vision-Language (V&L) models. We align the model with primitive visual attributes generated by Large Language Models (LLMs) Our method significantly outperforms current state-of-the-art prompt tuning methods on both novel class prediction and out-of-distribution generalization tasks.
  arXiv  Detail & Related papers  (2023-11-27T10:34:44Z)
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods [86.80718559904854]
  This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. We evaluate methods that use pretrained language models to produce descriptions of function behavior in natural language and code.
  arXiv  Detail & Related papers  (2023-09-07T17:47:26Z)
• Explaining Recommendation System Using Counterfactual Textual Explanations [4.318555434063274]
  It is found that if end-users understand the reason for the production of some output, it is easier to trust the system. One method for producing a more explainable output is using counterfactual reasoning.
  arXiv  Detail & Related papers  (2023-03-14T06:45:28Z)
• Getting the Most out of Simile Recognition [48.5838790615549]
  Simile recognition involves two subtasks: simile sentence classification that discriminates whether a sentence contains simile, and simile component extraction that locates the corresponding objects. Recent work ignores features other than surface strings. We study two types of features: 1) input-side features that include POS tags, dependency trees and word definitions, and 2) decoding features that capture the interdependence among various decoding decisions.
  arXiv  Detail & Related papers  (2022-11-11T03:22:45Z)
• SOInter: A Novel Deep Energy Based Interpretation Method for Explaining Structured Output Models [6.752231769293388]
  We propose a novel interpretation technique to explain the behavior of structured output models. We focus on one of the outputs as the target and try to find the most important features utilized by the structured model to decide on the target in each locality of the input space.
  arXiv  Detail & Related papers  (2022-02-20T21:57:07Z)
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions [55.660255727031725]
  Influence functions explain the decisions of a model by identifying influential training examples. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. We develop a new measure based on influence functions that can reveal artifacts in training data.
  arXiv  Detail & Related papers  (2020-05-14T00:45:23Z)
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer [89.21584915290319]
  We propose a self-attention attribution method to interpret the information interactions inside Transformer. We show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.
  arXiv  Detail & Related papers  (2020-04-23T14:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.