Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?

• URL: http://arxiv.org/abs/2607.17425v2
• Date: Tue, 21 Jul 2026 01:20:50 GMT
• Title: Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
• Abstract summary: We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs.<n>A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace.<n>These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse autoencoders (SAEs) compress model activations into sparse codes, but equal reconstruction error and sparsity can preserve different linearly decodable signals. We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs by combining this distortion with reconstruction loss. In a rank relaxation, an isotropic task prior saturates per-mode omission costs without changing PCA's ordering, whereas a structured prior can change which modes are retained. A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace. On GPT-2 small block 8, DPSAE reduces held-out decoder distortion by 10.6--11.4% across three paired runs while matching reconstruction NMSE. The same checkpoints pass an average natural-text output-KL noninferiority test, but one matched Pythia pair shows no improvement in probes restricted to a few sparse features. These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression, and that readout preservation is distinct from learning cleaner benchmark concepts or preserving every frozen-model behavior.

Related papers

• A Dominant Diffuse Phase in the Sparse Autoencoder Phase Diagram [0.0]
  Sparse autoencoders (SAEs) are increasingly used to recover interpretable features from neural-network activations.<n>The MAIS-O43 open problem proposes a controlled experiment to characterize when recovery of a true synthetic dictionary gives way to feature merging.
  arXiv  Detail & Related papers  (2026-09-09T15:13:42Z)
• Data Diversity, Not Frequency Invariance: A Controlled and Self-Audited Study of Compression-Robust Deepfake Detection [0.26856688022781555]
  Frequency features and compression-invariant representation learning are widely assumed to be key to deepfake detection that survives video compression.<n>We test this with CAFRL - block-DCT and FFT-phase streams, compression-level-conditioned band attention, and adversarial (gradient-reversal) compression.<n>Under a pre-registered protocol with capacity- and augmentation-matched controls, a plain EfficientNet-B0 on multi-quality data beat CAFRL as specified at every compression level.
  arXiv  Detail & Related papers  (2026-08-26T12:05:10Z)
• No More K-means: Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval [51.43543998583709]
  Single-stage Sparse Retrieval (SSR) is a paradigm shift that replaces expensive clustering with efficient sparse coding.<n>It reduces indexing time by 15x compared to ColBERTv2, halves retrieval latency, and simultaneously improves retrieval performance.
  arXiv  Detail & Related papers  (2026-05-28T15:53:34Z)
• Symbolic Regression via Latent Iterative Refinement [8.016092717336457]
  We propose Latent Equation Embedding (LEE), a framework that closes the gap between a one-shot prediction and the true posterior.<n>LEE learns a shared latent space Z equipped with three components.<n>LEE produces expressions 2--10x simpler than the strongest accuracy-oriented baselines.
  arXiv  Detail & Related papers  (2026-05-26T16:25:05Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Collapse-Free Prototype Readout Layer for Transformer Encoders [0.0]
  DDCL-Attention is a prototype-based readout layer for transformer encoders.<n>It produces compact token summaries at linear complexity in sequence length.<n>Same framework supports three uses: a final readout layer, a differentiable codebook extending VQ-VAE, and a hierarchical document compressor.
  arXiv  Detail & Related papers  (2026-04-04T20:23:21Z)
• GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networks [9.487330689135595]
  GRAIL is a simple zero-finetuning step applied after model compression.<n>It restores each block's input-output behavior using a small calibration set.<n>It consistently improves accuracy or perplexity over data-free and data-aware pruning or folding baselines.
  arXiv  Detail & Related papers  (2026-02-27T08:37:55Z)
• The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I) [6.453417258264177]
  This paper introduces Error-Bounded Predictive Coding ( EPC), a lossy text that leverages a Masked Language Model (MLM) as a decompressor.<n>Instead of storing a subset of original tokens, EPC allows the model to predict masked content and stores minimal, rank-based corrections only when the model's top prediction is incorrect.<n>We demonstrate that EPC consistently dominates Predictive Masking, offering superior fidelity at a significantly lower bit rate by more efficiently utilising the model's intrinsic knowledge.
  arXiv  Detail & Related papers  (2025-10-25T08:18:31Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)
• Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders [89.29256833403169]
  We introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods. KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.
  arXiv  Detail & Related papers  (2023-03-31T15:44:13Z)
• Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval [65.33981533521207]
  We introduce an approach to reduce predictive feature suppression for resource-constrained ICR methods: latent target decoding (LTD) LTD reconstructs the input caption in a latent space of a general-purpose sentence encoder, which prevents the image and caption encoder from suppressing predictive features. Our experiments show that, unlike reconstructing the input caption in the input space, LTD reduces predictive feature suppression, measured by obtaining higher recall@k, r-precision, and nDCG scores.
  arXiv  Detail & Related papers  (2022-04-28T09:55:28Z)
• Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization [93.95299500688286]
  We focus on prediction problems with structured outputs subject to output validity constraints. We propose composed fine-tuning, which fine-tunes a predictor composed with the pre-trained denoiser. For two-layer ReLU networks, we prove that composed fine-tuning significantly reduces the complexity of the predictor.
  arXiv  Detail & Related papers  (2020-06-29T17:14:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.