Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit

• URL: http://arxiv.org/abs/2512.10092v1
• Date: Wed, 10 Dec 2025 21:26:24 GMT
• Title: Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit
• Abstract summary: Analyzing large-scale text corpora is a core challenge in machine learning.<n>We propose using sparse autoencoders (SAEs) to create SAE embeddings.<n>We show that SAE embeddings are more cost-effective and reliable than LLMs and more controllable than dense embeddings.
• Score: 16.056849135589324
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Analyzing large-scale text corpora is a core challenge in machine learning, crucial for tasks like identifying undesirable model behaviors or biases in training data. Current methods often rely on costly LLM-based techniques (e.g. annotating dataset differences) or dense embedding models (e.g. for clustering), which lack control over the properties of interest. We propose using sparse autoencoders (SAEs) to create SAE embeddings: representations whose dimensions map to interpretable concepts. Through four data analysis tasks, we show that SAE embeddings are more cost-effective and reliable than LLMs and more controllable than dense embeddings. Using the large hypothesis space of SAEs, we can uncover insights such as (1) semantic differences between datasets and (2) unexpected concept correlations in documents. For instance, by comparing model responses, we find that Grok-4 clarifies ambiguities more often than nine other frontier models. Relative to LLMs, SAE embeddings uncover bigger differences at 2-8x lower cost and identify biases more reliably. Additionally, SAE embeddings are controllable: by filtering concepts, we can (3) cluster documents along axes of interest and (4) outperform dense embeddings on property-based retrieval. Using SAE embeddings, we study model behavior with two case studies: investigating how OpenAI model behavior has changed over time and finding "trigger" phrases learned by Tulu-3 (Lambert et al., 2024) from its training data. These results position SAEs as a versatile tool for unstructured data analysis and highlight the neglected importance of interpreting models through their data.

Related papers

• How Optimality Structures Sparse Dictionaries: A Theory for Understanding SAE Representations [0.8122270502556375]
  Sparse Autoencoders (SAEs) have found success parsing neural representations into interpretable concepts.<n>Here, we ask simply what properties any dictionary learning optimum must satisfy.<n>We use these constraints to explain a range of observed SAE behaviours.
  arXiv  Detail & Related papers  (2026-06-01T15:34:34Z)
• Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models [80.45129499188461]
  We introduce Qwen-Scope, an open-source suite of sparse autoencoders (SAEs) built on the Qwen model family.<n>We show that SAEs can go beyond post-hoc analysis to serve as practical interfaces for model development along four directions.
  arXiv  Detail & Related papers  (2026-05-12T10:01:06Z)
• Learning Compact Representations of LLM Abilities via Item Response Theory [35.74367665390977]
  We explore how to learn compact representations of large language models (LLMs)<n>We frame this problem as estimating the probability that a given model will correctly answer a specific query.<n>To learn these parameters jointly, we introduce a Mixture-of-Experts (MoE) network that couples model- and query-level embeddings.
  arXiv  Detail & Related papers  (2025-10-01T12:55:34Z)
• GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Models [0.0]
  Large language models (LLMs) are increasingly trained on massive, uncurated corpora.<n>We show that pairing LLMs with sparse autoencoders (SAEs) enables interpretation not only of model behavior but also of the deeper structures, themes, and biases embedded in the training data.<n>We train a GPT-style transformer model exclusively on the novels of Jane Austen, a corpus rich in social constructs and narrative patterns.
  arXiv  Detail & Related papers  (2025-09-24T11:10:16Z)
• ProtSAE: Disentangling and Interpreting Protein Language Models via Semantically-Guided Sparse Autoencoders [30.219733023958188]
  Sparse Autoencoder (SAE) has emerged as a powerful tool for mechanistic interpretability of large language models.<n>We propose a semantically-guided SAE, called ProtSAE.<n>We show that ProtSAE learns more biologically relevant and interpretable hidden features compared to previous methods.
  arXiv  Detail & Related papers  (2025-08-26T11:20:31Z)
• FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies [3.709351921096894]
  We propose FaithfulSAE, a method that trains SAEs on the model's own synthetic dataset.<n>We demonstrate that training SAEs on less-OOD instruction datasets results in SAEs being more stable across seeds.
  arXiv  Detail & Related papers  (2025-06-21T10:18:25Z)
• SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
  textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
  arXiv  Detail & Related papers  (2025-05-10T16:52:40Z)
• Sparse Autoencoders Trained on the Same Data Learn Different Features [0.7234862895932991]
  Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in large language models.<n>Our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features.
  arXiv  Detail & Related papers  (2025-01-28T01:24:16Z)
• SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models [88.29990536278167]
  We introduce SPaR, a self-play framework integrating tree-search self-refinement to yield valid and comparable preference pairs.<n>Our experiments show that a LLaMA3-8B model, trained over three iterations guided by SPaR, surpasses GPT-4-Turbo on the IFEval benchmark without losing general capabilities.
  arXiv  Detail & Related papers  (2024-12-16T09:47:43Z)
• Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge [55.65162959527848]
  Large language models have shown excellent performance on many knowledge-intensive tasks. However, pretraining data tends to contain misleading and even conflicting information. This study systematically analyze LLMs' learning preferences for data with conflicting knowledge.
  arXiv  Detail & Related papers  (2024-10-07T06:49:41Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science.<n>Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Interpretability at Scale: Identifying Causal Mechanisms in Alpaca [62.65877150123775]
  We use Boundless DAS to efficiently search for interpretable causal structure in large language models while they follow instructions. Our findings mark a first step toward faithfully understanding the inner-workings of our ever-growing and most widely deployed language models.
  arXiv  Detail & Related papers  (2023-05-15T17:15:40Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.