Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models

• URL: http://arxiv.org/abs/2605.15961v1
• Date: Fri, 15 May 2026 13:54:21 GMT
• Title: Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
• Abstract summary: Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks.<n>Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance.<n>We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model's visual representations.
• Score: 20.739509756066795
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale pre-trained vision-language models like CLIP demonstrate remarkable zero-shot performance across diverse tasks. However, fine-tuning these models to improve downstream performance often degrades robustness against distribution shifts. Recent approaches have attempted to mitigate this trade-off, but often rely on computationally expensive text-guidance. We propose a novel method for robust fine-tuning, SAE-FT, which operates only on the model's visual representations. SAE-FT regularizes changes to these representations by penalizing the addition and removal of semantically meaningful features identified by a Sparse Autoencoder trained on the pre-trained model. This constraint prevents catastrophic forgetting and makes the fine-tuning process interpretable, enabling direct analysis of semantic changes. SAE-FT is both mechanistically transparent and computationally efficient, matching or exceeding state-of-the-art performance on ImageNet and its associated distribution shift benchmarks. Code is publicly available at: https://github.com/Fabian-Mor/sae-ft.

Related papers

• Adaptive Depth Sparse Framework: Similarity-Driven Resource Allocation for Pre-Trained LLMs [18.457250148137494]
  We present an Adaptive Depth Sparse Framework (AdaDSF) that converts off-the-shelf pre-trained large language models into depth-sparse models without retraining.<n>On GPT-NeoX and Qwen2.5 over language modeling and commonsense reasoning, AdaDSF substantially reduces inference FLOPs while preserving performance close to dense counterparts.
  arXiv  Detail & Related papers  (2026-07-23T13:13:04Z)
• Interpretability Without Tradeoffs: Disentangling Polysemanticity At Equal Predictive Performance [58.49623467782041]
  ELUDe is a method for improving the interpretability of deep neural networks.<n>It breaks latent representations into clear, inspectable sub-units that behave like interpretable features.<n>It requires no explicit training, no labels, and can be applied to pretrained models.
  arXiv  Detail & Related papers  (2026-05-29T13:39:09Z)
• Concept-Guided Fine-Tuning: Steering ViTs away from Spurious Correlations to Improve Robustness [16.541207762213272]
  Vision Transformers (ViTs) often degrade under distribution shifts because they rely on spurious correlations.<n>We introduce a novel finetuning framework that steers model reasoning toward concept-level semantics.<n>Our method improves robustness across multiple ViT-based models.
  arXiv  Detail & Related papers  (2026-03-09T12:31:14Z)
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• EfficientFSL: Enhancing Few-Shot Classification via Query-Only Tuning in Vision Transformers [4.880377460177786]
  Large models such as Vision Transformers (ViTs) have demonstrated remarkable superiority over smaller architectures like ResNet in few-shot classification.<n>We propose EfficientFSL, a query-only fine-tuning framework tailored specifically for few-shot classification with ViT.<n>With minimal trainable parameters, EfficientFSL achieves state-of-the-art performance on four in-domain few-shot datasets and six cross-domain datasets.
  arXiv  Detail & Related papers  (2026-01-13T12:33:02Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment [70.87096576708898]
  We propose StarFT, a framework for fine-tuning zero-shot models to enhance robustness by preventing them from learning spuriosity.<n>StarFT boosts both worst-group and average accuracy by 14.30% and 3.02%, respectively, in the Waterbirds group shift scenario.
  arXiv  Detail & Related papers  (2025-05-19T15:15:35Z)
• TOAST: Transformer Optimization using Adaptive and Simple Transformations [40.311292704886235]
  We introduce TOAST, a framework that exploits redundancies to approximate entire transformer blocks with lightweight closed-form mappings.<n>Results show that large portions of transformer depth can be replaced by trivial functions, opening a new perspective on efficient foundation models.
  arXiv  Detail & Related papers  (2024-10-07T11:35:24Z)
• Masked Images Are Counterfactual Samples for Robust Fine-tuning [77.82348472169335]
  Fine-tuning deep learning models can lead to a trade-off between in-distribution (ID) performance and out-of-distribution (OOD) robustness. We propose a novel fine-tuning method, which uses masked images as counterfactual samples that help improve the robustness of the fine-tuning model.
  arXiv  Detail & Related papers  (2023-03-06T11:51:28Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.