HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation

• URL: http://arxiv.org/abs/2604.17751v1
• Date: Mon, 20 Apr 2026 03:11:01 GMT
• Title: HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation
• Abstract summary: Low-rank updates often concentrate energy on the leading singular directions of pretrained weights.<n>We propose HiP-LoRA, a spectrum-aware adaptation framework.<n>Under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adapting foundation models under resource budgets relies heavily on Parameter-Efficient Fine-Tuning (PEFT), with LoRA being a standard modular solution. However, LoRA suffers from spectral interference. Low-rank updates often concentrate energy on the leading singular directions of pretrained weights, perturbing general capabilities and causing catastrophic forgetting and fragile multi-adapter merging. To resolve this, we propose HiP-LoRA, a spectrum-aware adaptation framework. Utilizing the cached singular value decomposition (SVD) of pretrained layers, HiP-LoRA decomposes updates into two channels: a principal channel within the dominant singular subspace, and a residual low-rank channel in the orthogonal complement. A singular-value-weighted stability budget on the principal channel continuously balances pretrained behavior preservation with task-specific plasticity. Experiments on Llama-3.1-8B demonstrate that under matched budgets, HiP-LoRA drastically reduces pretraining degradation and multi-adapter MergeFail, robustly outperforming baselines in interference-sensitive tasks like continual tuning and knowledge editing.

Related papers

• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning [74.5532558466687]
  Group Relative Reward Rescaling (GR$3$) is a general, continuous, and reward-dependent gating mechanism.<n> GR$3$maintains training dynamics and downstream performance comparable to standard GRPO.<n>It significantly mitigates length inflation, outperforming state-of-the-art length-regularized baselines.
  arXiv  Detail & Related papers  (2026-03-11T08:41:34Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA [50.97792275353563]
  We introduce a novel framework that restructures a single Low-Rank Adaptation (LoRA) module as a decomposable Rank-1 Expert Pool.<n>Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [Guided] token.
  arXiv  Detail & Related papers  (2026-01-30T10:54:51Z)
• Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models [5.957171492626586]
  Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models.<n>We show that LoRA's vulnerability is fundamentally spectral.<n>Regularized Low-Rank Adaptation (RoRA) improves forgetting by increasing spectral strength.
  arXiv  Detail & Related papers  (2026-01-09T20:54:47Z)
• GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation [4.748720471060117]
  GRIT is a curvature-aware LoRA procedure that preserves the LoRA parameterization.<n>It matches or surpasses LoRA and QLoRA while reducing trainable parameters by 46% on average.<n>GRIT yields lower drift and a better updates-vs-retention frontier than strong PEFT-optimizer baselines.
  arXiv  Detail & Related papers  (2026-01-01T06:31:54Z)
• DoRAN: Stabilizing Weight-Decomposed Low-Rank Adaptation via Noise Injection and Auxiliary Networks [47.58150560549918]
  Weight-Decomposed Low-Rank Adaptation (DoRA) has been shown to improve both the learning capacity and training stability of the vanilla Low-Rank Adaptation (LoRA) method.<n>We propose DoRAN, a new variant of DoRA designed to further stabilize training and boost the sample efficiency of DoRA.
  arXiv  Detail & Related papers  (2025-10-05T19:27:48Z)
• Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes [9.4848188271008]
  Low-Rank Adaptation (LoRA) has proven effective in reducing computational costs while maintaining performance comparable to fully fine-tuned foundation models.<n>Current adaptive LoRA methods attempt to overcome this limitation by dynamically expanding or selectively allocating ranks.<n>We introduce Stable Rank-Guided Low-Rank Adaptation (SR-LoRA), a novel framework that utilizes the stable rank of pre-trained weight matrices as a natural prior for layer-wise rank allocation.
  arXiv  Detail & Related papers  (2025-06-30T23:54:23Z)
• SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization [2.594346658179846]
  Low-Rank Adaptation (LoRA) constrains updates to a fixed low-rank subspace.<n>We introduce Subspace Recomposition in Low-Rank Adaptation (SRLoRA) via importance-based fusion and reinitialization.<n> SRLoRA consistently achieves faster convergence and improved accuracy over standard LoRA.
  arXiv  Detail & Related papers  (2025-05-18T14:12:40Z)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning [73.93639228235622]
  Continual Learning with foundation models has emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks.<n>Existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks.<n>We propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal.
  arXiv  Detail & Related papers  (2025-01-22T20:00:41Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.