Do Sparse Autoencoders Generalize? A Case Study of Answerability

• URL: http://arxiv.org/abs/2502.19964v1
• Date: Thu, 27 Feb 2025 10:45:25 GMT
• Title: Do Sparse Autoencoders Generalize? A Case Study of Answerability
• Authors: Lovis Heindrich, Philip Torr, Fazl Barez, Veronika Thost,
• Abstract summary: We evaluate SAE feature generalization across diverse answerability datasets for Gemma 2 SAEs.<n>Our analysis reveals that residual stream probes outperform SAE features within domains, but generalization performance differs sharply.
• Score: 12.131254862319865
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse autoencoders (SAEs) have emerged as a promising approach in language model interpretability, offering unsupervised extraction of sparse features. For interpretability methods to succeed, they must identify abstract features across domains, and these features can often manifest differently in each context. We examine this through "answerability"-a model's ability to recognize answerable questions. We extensively evaluate SAE feature generalization across diverse answerability datasets for Gemma 2 SAEs. Our analysis reveals that residual stream probes outperform SAE features within domains, but generalization performance differs sharply. SAE features demonstrate inconsistent transfer ability, and residual stream probes similarly show high variance out of distribution. Overall, this demonstrates the need for quantitative methods to predict feature generalization in SAE-based interpretability.

Related papers

• FADE: Why Bad Descriptions Happen to Good Features [14.00042287629001]
  We introduce FADE: Feature Alignment to Description Evaluation.<n>FADE is a scalable framework for evaluating feature-description alignment.<n>We apply FADE to analyze existing open-source feature descriptions, and assess key components of automated interpretability pipelines.
  arXiv  Detail & Related papers  (2025-02-24T09:28:35Z)
• Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition [8.950906917573986]
  Few-shot adaptation for Vision-Language Models (VLMs) presents a dilemma: balancing in-distribution accuracy with out-of-distribution generalization.<n>Recent research has utilized low-level concepts such as visual attributes to enhance generalization.<n>This study reveals that VLMs overly rely on a small subset of attributes on decision-making, which co-occur with the category but are not inherently part of it, spuriously correlated attributes.
  arXiv  Detail & Related papers  (2025-02-19T12:05:33Z)
• Sparse Autoencoder Features for Classifications and Transferability [11.2185030332009]
  We analyze Sparse Autoencoders (SAEs) for interpretable feature extraction from Large Language Models (LLMs)<n>Our framework evaluates (1) model-layer selection and scaling properties, (2) SAE architectural configurations, including width and pooling strategies, and (3) the effect of binarizing continuous SAE activations.
  arXiv  Detail & Related papers  (2025-02-17T02:30:45Z)
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders [115.34050914216665]
  Sparse Autoencoders (SAEs) have emerged as a powerful unsupervised method for extracting sparse representations from language models. We introduce a suite of 256 SAEs, trained on each layer and sublayer of the Llama-3.1-8B-Base model, with 32K and 128K features. We assess the generalizability of SAEs trained on base models to longer contexts and fine-tuned models.
  arXiv  Detail & Related papers  (2024-10-27T17:33:49Z)
• Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation [41.66053021998106]
  Aspect-based sentiment analysis (ABSA) is a challenging task.<n>Traditional evaluation methods often constrain ground truths (GT) to a single term.<n>We propose a novel and fully automated pipeline that expands existing evaluation sets by adding alternative valid terms for aspect and opinion.
  arXiv  Detail & Related papers  (2024-10-13T11:48:09Z)
• A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders [0.0]
  Sparse Autoencoders (SAEs) have emerged as a promising approach to decompose the activations of Large Language Models (LLMs) In this paper, we pose two questions. First, to what extent do SAEs extract monosemantic and interpretable latents? Second, to what extent does varying the sparsity or the size of the SAE affect monosemanticity / interpretability?
  arXiv  Detail & Related papers  (2024-09-22T16:11:02Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context. We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions. We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Diffexplainer: Towards Cross-modal Global Explanations with Diffusion Models [51.21351775178525]
  DiffExplainer is a novel framework that, leveraging language-vision models, enables multimodal global explainability. It employs diffusion models conditioned on optimized text prompts, synthesizing images that maximize class outputs. The analysis of generated visual descriptions allows for automatic identification of biases and spurious features.
  arXiv  Detail & Related papers  (2024-04-03T10:11:22Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction [67.54420015049732]
  Aspect Sentiment Triplet Extraction (ASTE) is a challenging task in sentiment analysis, aiming to provide fine-grained insights into human sentiments. Existing benchmarks are limited to two domains and do not evaluate model performance on unseen domains. We introduce a domain-expanded benchmark by annotating samples from diverse domains, enabling evaluation of models in both in-domain and out-of-domain settings.
  arXiv  Detail & Related papers  (2023-05-23T18:01:49Z)
• Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition [59.52434325897716]
  We propose a solution, named DMUE, to address the problem of annotation ambiguity from two perspectives. For the former, an auxiliary multi-branch learning framework is introduced to better mine and describe the latent distribution in the label space. For the latter, the pairwise relationship of semantic feature between instances are fully exploited to estimate the ambiguity extent in the instance space.
  arXiv  Detail & Related papers  (2021-04-01T03:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.