Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning

• URL: http://arxiv.org/abs/2607.09287v1
• Date: Fri, 10 Jul 2026 10:55:28 GMT
• Title: Super-Tuning: From Activation-Aware Pruning to Sparse Fine-Tuning
• Abstract summary: We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt.<n>We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support.<n>We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA.
• Score: 47.97857843162072
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) remain expensive to fine-tune because full-parameter updates require substantial memory, compute, and per-task storage. We study whether saliency signals originally developed for pruning can be reused to choose where a model should adapt. We propose Super, a sparse parameter-efficient fine-tuning (PEFT) method that fixes a small trainable support using a Wanda-style activation-weighted magnitude score [Sun et al., 2023] computed from a calibration pass. We then introduce Supra, a hybrid adapter that combines this sparse update with LoRA while preserving a matched trainable-parameter budget through a simple budget-splitting rule. In single-seed Math17K arithmetic experiments on Llama-3.2-1B and Meta-Llama-3-8B, the best Super/Supra variants achieve the highest average accuracy among the tested schedule-selected adapter configurations. We also include a PaFi-style magnitude-only support as a closest training-free sparse baseline and find that low-score supports under both magnitude and Wanda-style orderings can be effective. These results suggest that simple pruning-inspired orderings can provide useful fixed sparse supports for PEFT, especially when combined with low-rank adapters.

Related papers

• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• Rethinking Adapter Placement: A Dominant Adaptation Module Perspective [14.452224342067657]
  Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning method that places low-rank adapters into frozen pre-trained models.<n>Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly.<n>We propose DomLoRA, a placement method that places a single adapter at the dominant adaptation module.
  arXiv  Detail & Related papers  (2026-05-07T13:01:00Z)
• Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models [19.288371639304504]
  Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs.<n>We introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning.
  arXiv  Detail & Related papers  (2026-01-08T20:34:12Z)
• FLoRA: Fused forward-backward adapters for parameter efficient fine-tuning and reducing inference-time latencies of LLMs [7.771813594229729]
  We propose a family of fused forward-backward adapters (FFBA) for parameter-efficient fine-tuning of large language models (LLMs) on downstream tasks.<n> Experimental results show that the proposed FFB adapters perform significantly better than the popularly used LoRA in both accuracy and latency.
  arXiv  Detail & Related papers  (2025-10-28T12:45:45Z)
• Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning [82.16625951603315]
  NoRA replaces fixed activations with learnable rational functions and applies structured low-rank updates to numerator and denominator coefficients.<n>On vision transformers trained on CIFAR-10 and CIFAR-100, NoRA matches or exceeds full fine-tuning while updating only 0.4% of parameters.<n>NoRA constrains adaptation to a low-dimensional functional subspace, implicitly regularizing update magnitude and direction.
  arXiv  Detail & Related papers  (2025-09-16T16:47:03Z)
• Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts [72.22148263683037]
  We study the properties of sparse adapters, which train only a subset of weights in the base neural network, as potential building blocks of modular architectures.<n>First, we propose a simple method for training highly effective sparse adapters, which is conceptually simpler than existing methods in the literature.<n>Next, we investigate the merging properties of these sparse adapters by merging adapters for up to 20 natural language processing tasks.
  arXiv  Detail & Related papers  (2025-07-09T03:25:45Z)
• Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models [108.08773541490191]
  Pre-trained Language models (PLMs) have a huge amount of parameters, fine-tuning them is often expensive and time consuming. It is necessary to adopt a parameter-efficient approach to reduce parameters of PLMs in fine-tuning without compromising their performance in downstream tasks. In this paper, we design a novel adapter which only acts on self-attention outputs in PLMs.
  arXiv  Detail & Related papers  (2024-07-04T18:21:28Z)
• RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation [30.797422827190278]
  We present a new PEFT method called Robust Adaptation (RoSA) inspired by robust principal component analysis. RoSA trains $textitlow-rank$ and $textithighly-sparse$ components on top of a set of fixed pretrained weights. We show that RoSA outperforms LoRA, pure sparse fine-tuning, and alternative hybrid methods at the same parameter budget.
  arXiv  Detail & Related papers  (2024-01-09T17:09:01Z)
• LoRA: Low-Rank Adaptation of Large Language Models [71.75808607987281]
  Low-Rank Adaptation, or LoRA, freezes the pre-trained model weights and injects trainable rank decomposition into each layer of the Transformer architecture. For GPT-3, LoRA can reduce the number of trainable parameters by 10,000 times and the computation hardware requirement by 3 times compared to full fine-tuning.
  arXiv  Detail & Related papers  (2021-06-17T17:37:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.