AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers

• URL: http://arxiv.org/abs/2603.14706v1
• Date: Mon, 16 Mar 2026 01:29:26 GMT
• Title: AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers
• Abstract summary: We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck.<n>On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average.<n>Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested.
• Score: 1.0152838128195467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune

Related papers

• TALAN: Task-Aligned Latent Adaptation Networks for Targeted Post-Training of Large Language Models [8.47008876232157]
  We introduce TALAN (Task-Aligned Latent Adaptation Networks), a sequence-conditioned latent side path inserted into a transformer's residual stream.<n> TALAN compresses the active sequence into latent memory, remixes it into token-level perturbations, and writes them back through a controlled residual update.<n>Across four Qwen3-family backbones and four STEM/code benchmarks, TALAN improves matched LoRA and DoRA baselines.
  arXiv  Detail & Related papers  (2026-06-05T04:34:50Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• Correcting Suppressed Log-Probabilities in Language Models with Post-Transformer Adapters [0.0]
  Post-transformer adapter corrects suppression on 31 ideology-discriminating facts across Qwen3-4B, 8B, and 14B.<n> adapter memorizes all 15 training facts and generalizes to 11--39% of 16 held-out facts across 5 random splits per scale.
  arXiv  Detail & Related papers  (2026-03-25T16:27:43Z)
• Low-Rank Adaptation Secretly Imitates Differentially Private SGD [5.359060261460183]
  We show theoretically that low-rank adaptation is equivalent to fine-tuning adapters with noisy batch gradients.<n>We also quantify the variance of the injected noise as a decreasing function of adaptation rank.<n>Low-rank adaptation provides robustness to membership inference attacks w.r.t the fine-tuning data.
  arXiv  Detail & Related papers  (2024-09-26T04:56:49Z)
• Parameter-Efficient Sparse Retrievers and Rerankers using Adapters [4.9545244468634655]
  We study adapters for SPLADE, a sparse retriever, for which adapters retain the efficiency and effectiveness otherwise achieved by finetuning. We also address domain adaptation of neural retrieval thanks to adapters on cross-domain BEIR datasets and TripClick.
  arXiv  Detail & Related papers  (2023-03-23T12:34:30Z)
• Revisiting adapters with adversarial training [39.018061382248085]
  We show that it is not necessary to separate batch statistics when co-training on clean and adversarial inputs. We show that training with adapters enables model soups through linear combinations of the clean and adversarial tokens.
  arXiv  Detail & Related papers  (2022-10-10T17:58:14Z)
• SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters [96.52807311742198]
  We re-examine the parameter-efficiency of Adapters through the lens of network pruning. We find that SparseAdapter can achieve comparable or better performance than standard Adapters when the sparse ratio reaches up to 80%.
  arXiv  Detail & Related papers  (2022-10-09T15:28:48Z)
• AdaMix: Mixture-of-Adapter for Parameter-efficient Tuning of Large Language Models [119.7093605087114]
  Fine-tuning large-scale pre-trained language models to downstream tasks require updating hundreds of millions of parameters. This not only increases the serving cost to store a large copy of the model weights for every task, but also exhibits instability during few-shot task adaptation. We introduce a new mechanism to improve adapter capacity without increasing parameters or computational cost by two key techniques.
  arXiv  Detail & Related papers  (2022-05-24T23:41:22Z)
• AdapterBias: Parameter-efficient Token-dependent Representation Shift for Adapters in NLP Tasks [55.705355299065474]
  Transformer-based pre-trained models with millions of parameters require large storage. Recent approaches tackle this shortcoming by training adapters, but these approaches still require a relatively large number of parameters. In this study, AdapterBias, a surprisingly simple yet effective adapter architecture, is proposed.
  arXiv  Detail & Related papers  (2022-04-30T16:49:41Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)
• AdapterDrop: On the Efficiency of Adapters in Transformers [53.845909603631945]
  Massively pre-trained transformer models are computationally expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters.
  arXiv  Detail & Related papers  (2020-10-22T17:49:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.