How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models

• URL: http://arxiv.org/abs/2603.25325v1
• Date: Thu, 26 Mar 2026 11:12:42 GMT
• Title: How Pruning Reshapes Features: Sparse Autoencoder Analysis of Weight-Pruned Language Models
• Abstract summary: We present the first systematic study of how unstructured pruning reshapes the feature geometry of language models.<n>We investigate five research questions spanning seed stability, feature survival, SAE transferability, feature fragility, and causal relevance.<n>Our most striking finding is that rare SAE features--those with low firing rates--survive pruning far better than frequent ones.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weight pruning is a standard technique for compressing large language models, yet its effect on learned internal representations remains poorly understood. We present the first systematic study of how unstructured pruning reshapes the feature geometry of language models, using Sparse Autoencoders (SAEs) as interpretability probes. Across three model families (Gemma 3 1B, Gemma 2 2B, Llama 3.2 1B), two pruning methods (magnitude and Wanda), and six sparsity levels (0--60%), we investigate five research questions spanning seed stability, feature survival, SAE transferability, feature fragility, and causal relevance. Our most striking finding is that rare SAE features--those with low firing rates--survive pruning far better than frequent ones, with within-condition Spearman correlations of rho = -1.0 in 11 of 17 experimental conditions. This counter-intuitive result suggests that pruning acts as implicit feature selection, preferentially destroying high-frequency generic features while preserving specialized rare ones. We further show that Wanda pruning preserves feature structure up to 3.7x better than magnitude pruning, that pre-trained SAEs remain viable on Wanda-pruned models up to 50% sparsity, and that geometric feature survival does not predict causal importance--a dissociation with implications for interpretability under compression.

Related papers

• When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs [15.418870425179875]
  Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models.<n>We show that for a fixed SAE, its impact is governed by perturbation energy.<n>We propose a layer-wise sparsity allocation strategy, achieving lower perplexity under the same average pruning sparsity.
  arXiv  Detail & Related papers  (2026-08-26T15:57:42Z)
• Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects [1.630081999385542]
  Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison.<n>We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth.
  arXiv  Detail & Related papers  (2026-07-22T17:33:16Z)
• From Sparse Features to Trustworthy Proxies: Certifying SAE-Based Interpretability [53.681678236115836]
  We study when can an SAE-based explanation be treated as a faithful view of an underlying frozen LM.<n>Our framework derives an upper bound on the base model's expected risk using four measurable quantities.<n>We show that the bound becomes non-vacuous on GPT-2 Small, Gemma-2B, and Llama-3-8B at practical sample sizes.
  arXiv  Detail & Related papers  (2026-06-16T18:28:23Z)
• Perplexity Can Miss SAE Feature Damage Under Quantization [0.16921396880325779]
  Quantized models are judged acceptable when perplexity or downstream accuracy remains close to the full-precision original.<n>We test this by using a frozen sparse-autoencoder (SAE) as a fixed measurement basis.
  arXiv  Detail & Related papers  (2026-06-02T01:17:05Z)
• Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models [80.45129499188461]
  We introduce Qwen-Scope, an open-source suite of sparse autoencoders (SAEs) built on the Qwen model family.<n>We show that SAEs can go beyond post-hoc analysis to serve as practical interfaces for model development along four directions.
  arXiv  Detail & Related papers  (2026-05-12T10:01:06Z)
• Fundamental Limits of Neural Network Sparsification: Evidence from Catastrophic Interpretability Collapse [7.167095059974211]
  This work investigates feature survival under severe capacity constraints in Variational Autoencoder--Sparse Autoencoder architectures.<n>We introduce an adaptive sparsity scheduling framework that progressively reduces active neurons from 500 to 50 over 50 training epochs.<n>Our key finding reveals a pervasive paradox: while global representation quality remains stable, local feature interpretability collapses systematically.
  arXiv  Detail & Related papers  (2026-03-18T00:16:38Z)
• Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines? [10.871959954490217]
  Sparse Autoencoders (SAEs) have emerged as a promising tool for interpreting neural networks by decomposing their activations into sparse sets of human-interpretable features.<n>Recent work has introduced multiple SAE variants and successfully scaled them to frontier models.<n>Despite much excitement, a growing number of negative results in downstream tasks casts doubt on whether SAEs recover meaningful features.
  arXiv  Detail & Related papers  (2026-02-15T11:53:55Z)
• Sparse Semantic Dimension as a Generalization Certificate for LLMs [53.681678236115836]
  We introduce the Sparse Semantic Dimension (SSD), a complexity measure derived from the active feature vocabulary of a Sparse Autoencoder (SAE) trained on the model's layers.<n>We validate this framework on GPT-2 Small and Gemma-2B, demonstrating that our bound provides non-vacuous certificates at realistic sample sizes.
  arXiv  Detail & Related papers  (2026-02-11T21:45:18Z)
• PolySAE: Modeling Feature Interactions in Sparse Autoencoders via Polynomial Decoding [16.10987386200554]
  We introduce PolySAE, which extends the SAE decoder with higher-order terms to model feature interactions.<n>Across four language models and three SAE variants, PolySAE achieves an average improvement of approximately 8% in probing F1.
  arXiv  Detail & Related papers  (2026-02-01T16:34:45Z)
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training [86.64715217940274]
  Outliers function jointly with normalization.<n>Outliers serve more as rescale factors rather than contributors.<n>Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling.
  arXiv  Detail & Related papers  (2026-01-30T13:29:45Z)
• Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders [63.544453925182005]
  We train 90 SAEs across three language models and evaluate their interpretability and steering utility.<n>Our analysis reveals only a relatively weak positive association (tau b approx 0.298), indicating that interpretability is an insufficient proxy for steering performance.<n>We propose a novel selection criterion called Delta Token Confidence, which measures how much amplifying a feature changes the next token distribution.
  arXiv  Detail & Related papers  (2025-10-04T04:14:50Z)
• NIRVANA: Structured pruning reimagined for large language models compression [50.651730342011014]
  We introduce NIRVANA, a novel pruning method designed to balance immediate zero-shot preservation accuracy with robust fine-tuning.<n>To further address the unique challenges posed by structured pruning, NIRVANA incorporates an adaptive sparsity allocation mechanism across layers and modules.<n>Experiments conducted on Llama3, Qwen, T5 models demonstrate that NIRVANA outperforms existing structured pruning methods under equivalent sparsity constraints.
  arXiv  Detail & Related papers  (2025-09-17T17:59:00Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Sparse Autoencoders Trained on the Same Data Learn Different Features [0.7234862895932991]
  Sparse autoencoders (SAEs) are a useful tool for uncovering human-interpretable features in large language models.<n>Our research shows that SAEs trained on the same model and data, differing only in the random seed used to initialize their weights, identify different sets of features.
  arXiv  Detail & Related papers  (2025-01-28T01:24:16Z)
• Explicit Tradeoffs between Adversarial and Natural Distributional Robustness [48.44639585732391]
  In practice, models need to enjoy both types of robustness to ensure reliability. In this work, we show that in fact, explicit tradeoffs exist between adversarial and natural distributional robustness.
  arXiv  Detail & Related papers  (2022-09-15T19:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.