Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders

• URL: http://arxiv.org/abs/2510.03659v1
• Date: Sat, 04 Oct 2025 04:14:50 GMT
• Title: Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders
• Authors: Xu Wang, Yan Hu, Benyou Wang, Difan Zou,
• Abstract summary: We train 90 SAEs across three language models and evaluate their interpretability and steering utility.<n>Our analysis reveals only a relatively weak positive association (tau b approx 0.298), indicating that interpretability is an insufficient proxy for steering performance.<n>We propose a novel selection criterion called Delta Token Confidence, which measures how much amplifying a feature changes the next token distribution.
• Score: 63.544453925182005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Autoencoders (SAEs) are widely used to steer large language models (LLMs), based on the assumption that their interpretable features naturally enable effective model behavior steering. Yet, a fundamental question remains unanswered: does higher interpretability indeed imply better steering utility? To answer this question, we train 90 SAEs across three LLMs (Gemma-2-2B, Qwen-2.5-3B, Gemma-2-9B), spanning five architectures and six sparsity levels, and evaluate their interpretability and steering utility based on SAEBench (arXiv:2501.12345) and AxBench (arXiv:2502.23456) respectively, and perform a rank-agreement analysis via Kendall's rank coefficients (tau b). Our analysis reveals only a relatively weak positive association (tau b approx 0.298), indicating that interpretability is an insufficient proxy for steering performance. We conjecture the interpretability utility gap may stem from the selection of SAE features, as not all of them are equally effective for steering. To further find features that truly steer the behavior of LLMs, we propose a novel selection criterion called Delta Token Confidence, which measures how much amplifying a feature changes the next token distribution. We show that our method improves the steering performance of three LLMs by 52.52 percent compared to the current best output score based criterion (arXiv:2503.34567). Strikingly, after selecting features with high Delta Token Confidence, the correlation between interpretability and utility vanishes (tau b approx 0), and can even become negative. This further highlights the divergence between interpretability and utility for the most effective steering features.

Related papers

• Mechanistic Indicators of Steering Effectiveness in Large Language Models [3.635648354808971]
  Activation-based steering enables Large Language Models to exhibit targeted behaviors by intervening on intermediate activations without retraining.<n>Despite its widespread use, the mechanistic factors that govern when steering succeeds or fails remain poorly understood.<n>We investigate whether the reliability of steering can be diagnosed using internal model signals.
  arXiv  Detail & Related papers  (2026-02-02T06:56:22Z)
• Structured Uncertainty guided Clarification for LLM Agents [126.26213027785813]
  LLM agents extend large language models with tool-calling capabilities, but ambiguous user instructions often lead to incorrect invocations and task failures.<n>We introduce a principled formulation of structured uncertainty over tool-call parameters, modeling joint tool-argument clarification as a POMDP with Expected Value of Perfect Information (EVPI) objective for optimal question selection and aspect-based cost modeling to prevent redundancy.<n>Our SAGE-Agent leverages this structured uncertainty to achieve superior efficiency: increasing coverage on ambiguous tasks by 7-39% while reducing clarification questions by 1.5-2.7$times$ compared to strong prompting and uncertainty-based baselines.
  arXiv  Detail & Related papers  (2025-11-11T21:50:44Z)
• A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering [0.0]
  We propose focusing on a single, most relevant SAE latent (top-1), eliminating redundant features.<n>We show that steering an SAE latent associated with reasoning reliably elicits step-by-step mathematical reasoning.<n>Our results demonstrate that SAEs outperform mean activation difference methods on mathematical reasoning benchmarks and match their performance on IF-Eval.
  arXiv  Detail & Related papers  (2025-09-24T08:31:31Z)
• Personality as a Probe for LLM Evaluation: Method Trade-offs and Downstream Effects [0.6087817758152709]
  We present a systematic study of personality control using the Big Five traits.<n>Trait-level analysis shows openness as uniquely challenging, agreeableness as most resistant to ICL.<n>Experiments on Gemma-2-2B-IT and LLaMA-3-8B-Instruct reveal clear trade-offs.
  arXiv  Detail & Related papers  (2025-09-05T04:19:15Z)
• CorrSteer: Steering Improves Task Performance and Safety in LLMs through Correlation-based Sparse Autoencoder Feature Selection [0.0]
  We propose CorrSteer, which selects features by correlating sample correctness with SAE activations from generated tokens at inference time.<n>Our work establishes correlationbased selection as an effective and scalable approach for automated SAE steering across language model applications.
  arXiv  Detail & Related papers  (2025-08-18T00:01:42Z)
• Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models [48.40096116617163]
  Large Language Models (LLMs) demonstrate the ability to solve reasoning and mathematical problems using the Chain-of-Thought (CoT) technique.<n>This work, inspired by the deep thinking paradigm of DeepSeek-R1, utilizes a steering technique to enhance the reasoning ability of an LLM without external datasets.
  arXiv  Detail & Related papers  (2025-05-21T15:17:59Z)
• SEAL: Steerable Reasoning Calibration of Large Language Models for Free [58.190800043449336]
  Large Language Models (LLMs) have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism.<n>Recent studies reveal substantial redundancy in the CoT reasoning traces, which negatively impacts model performance.<n>We introduce SEAL, a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains.
  arXiv  Detail & Related papers  (2025-04-07T02:42:07Z)
• AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders [73.37603699731329]
  We introduce AxBench, a large-scale benchmark for steering and concept detection.<n>For steering, we find that prompting outperforms all existing methods, followed by finetuning.<n>For concept detection, representation-based methods such as difference-in-means, perform the best.
  arXiv  Detail & Related papers  (2025-01-28T18:51:24Z)
• Tailoring Self-Rationalizers with Multi-Reward Distillation [88.95781098418993]
  Large language models (LMs) are capable of generating free-text rationales to aid question answering. In this work, we enable small-scale LMs to generate rationales that improve downstream task performance. Our method, MaRio, is a multi-reward conditioned self-rationalization algorithm.
  arXiv  Detail & Related papers  (2023-11-06T00:20:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.