Sparse Autoencoders Trained on the Same Data Learn Different Features

• URL: http://arxiv.org/abs/2501.16615v2
• Date: Wed, 29 Jan 2025 19:18:45 GMT
• Title: Sparse Autoencoders Trained on the Same Data Learn Different Features
• Authors: Gonçalo Paulo, Nora Belrose,
• Abstract summary: Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in large language models. Our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features.
• Score: 0.7234862895932991
• License:
• Abstract: Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in the activations of large language models (LLMs). While some expect SAEs to find the true underlying features used by a model, our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features. For example, in an SAE with 131K latents trained on a feedforward network in Llama 3 8B, only 30% of the features were shared across different seeds. We observed this phenomenon across multiple layers of three different LLMs, two datasets, and several SAE architectures. While ReLU SAEs trained with the L1 sparsity loss showed greater stability across seeds, SAEs using the state-of-the-art TopK activation function were more seed-dependent, even when controlling for the level of sparsity. Our results suggest that the set of features uncovered by an SAE should be viewed as a pragmatically useful decomposition of activation space, rather than an exhaustive and universal list of features "truly used" by the model.

Related papers

• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models [88.29990536278167]
  We introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs free from distractions. Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
  arXiv  Detail & Related papers  (2024-12-16T09:47:43Z)
• Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders [8.003244901104111]
  We propose a regularization technique for improving feature learning by encouraging SAEs trained in parallel to learn similar features. textscMFR can improve the reconstruction loss of SAEs by up to 21.21% on GPT-2 Small, and 6.67% on EEG data.
  arXiv  Detail & Related papers  (2024-11-02T11:42:23Z)
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders [115.34050914216665]
  Sparse Autoencoders (SAEs) have emerged as a powerful unsupervised method for extracting sparse representations from language models. We introduce a suite of 256 SAEs, trained on each layer and sublayer of the Llama-3.1-8B-Base model, with 32K and 128K features. We assess the generalizability of SAEs trained on base models to longer contexts and fine-tuned models.
  arXiv  Detail & Related papers  (2024-10-27T17:33:49Z)
• Automatically Interpreting Millions of Features in Large Language Models [1.8035046415192353]
  sparse autoencoders (SAEs) can be used to transform activations into a higher-dimensional latent space. We build an open-source pipeline to generate and evaluate natural language explanations for SAE features. Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons.
  arXiv  Detail & Related papers  (2024-10-17T17:56:01Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science. Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning [0.9374652839580183]
  Identifying the features learned by neural networks is a core challenge in mechanistic interpretability. We propose end-to-end sparse dictionary learning, a method for training SAEs that ensures the features learned are functionally important. We explore geometric and qualitative differences between e2e SAE features and standard SAE features.
  arXiv  Detail & Related papers  (2024-05-17T17:03:46Z)
• Self-Supervised Learning for Invariant Representations from Multi-Spectral and SAR Images [5.994412766684843]
  Self-Supervised learning (SSL) has become the new state-of-art in several domain classification and segmentation tasks. This work proposes RSDnet, which applies the distillation network (BYOL) in the remote sensing (RS) domain.
  arXiv  Detail & Related papers  (2022-05-04T13:16:48Z)
• Lightweight Single-Image Super-Resolution Network with Attentive Auxiliary Feature Learning [73.75457731689858]
  We develop a computation efficient yet accurate network based on the proposed attentive auxiliary features (A$2$F) for SISR. Experimental results on large-scale dataset demonstrate the effectiveness of the proposed model against the state-of-the-art (SOTA) SR methods.
  arXiv  Detail & Related papers  (2020-11-13T06:01:46Z)
• Adversarial Feature Hallucination Networks for Few-Shot Learning [84.31660118264514]
  Adversarial Feature Hallucination Networks (AFHN) is based on conditional Wasserstein Generative Adversarial networks (cWGAN) Two novel regularizers are incorporated into AFHN to encourage discriminability and diversity of the synthesized features.
  arXiv  Detail & Related papers  (2020-03-30T02:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.