Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps

• URL: http://arxiv.org/abs/2506.16787v1
• Date: Fri, 20 Jun 2025 07:09:05 GMT
• Title: Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
• Abstract summary: Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models.<n>Despite its successes, the substantial parameter redundancy, which limits the capacity and efficiency of LoRA, has been recognized as a bottleneck.<n>We introduce underlineSpectral-underlineencoding underlineLow-underlineRank underlineAdaptation (SeLoRA)
• Score: 37.00767702936744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-Rank Adaptation (LoRA) has emerged as a prominent technique for fine-tuning large foundation models. Despite its successes, the substantial parameter redundancy, which limits the capacity and efficiency of LoRA, has been recognized as a bottleneck. In this work, we systematically investigate the impact of redundancy in fine-tuning LoRA and reveal that reducing density redundancy does not degrade expressiveness. Based on this insight, we introduce \underline{S}pectral-\underline{e}ncoding \underline{L}ow-\underline{R}ank \underline{A}daptation (SeLoRA), which harnesses the robust expressiveness of spectral bases to re-parameterize LoRA from a sparse spectral subspace. Designed with simplicity, SeLoRA enables seamless integration with various LoRA variants for performance boosting, serving as a scalable plug-and-play framework. Extensive experiments substantiate that SeLoRA achieves greater efficiency with fewer parameters, delivering superior performance enhancements over strong baselines on various downstream tasks, including commonsense reasoning, math reasoning, and code generation.

Related papers

• CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA [28.777366440905382]
  Low-Rank Adaptation (LoRA) is one of the most widely adopted parameter-efficient fine-tuning methods.<n>Care-LoRA replaces the full input activation with the low-rank compressed activation naturally produced by the LoRA branch.<n>Care-LoRA achieves competitive or even superior performance compared with standard LoRA and representative LoRA variants.
  arXiv  Detail & Related papers  (2026-07-11T09:04:58Z)
• Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA [50.97792275353563]
  We introduce a novel framework that restructures a single Low-Rank Adaptation (LoRA) module as a decomposable Rank-1 Expert Pool.<n>Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [Guided] token.
  arXiv  Detail & Related papers  (2026-01-30T10:54:51Z)
• Latent Space Factorization in LoRA [10.994747174370099]
  Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning.<n>We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA) to learn two distinct latent spaces.<n>Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information.
  arXiv  Detail & Related papers  (2025-10-22T14:37:20Z)
• TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptation [19.91254891769464]
  LoRA has become one of the most widely used parameter-efficient fine-tuning methods due to its simplicity and effectiveness.<n>Numerous studies have shown that LoRA often introduces substantial parameter redundancy, which not only increases the number of trainable parameters but also hinders the effectiveness of fine-tuning.<n>We propose TASO, a redundancy reduction method that leverages importance information from the pretrained model's weights to mitigate LoRA redundancy.
  arXiv  Detail & Related papers  (2025-09-22T12:29:43Z)
• Uni-LoRA: One Vector is All You Need [13.938834666101679]
  Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models.<n>In this paper, we show that the parameter space reduction strategies employed by these LoRA variants can be formulated within a unified framework.<n>Under the unified view of Uni-LoRA, this design requires only a single trainable vector to reconstruct LoRA parameters for the entire LLM.
  arXiv  Detail & Related papers  (2025-06-01T03:00:09Z)
• SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization [2.594346658179846]
  Low-Rank Adaptation (LoRA) constrains updates to a fixed low-rank subspace.<n>We introduce Subspace Recomposition in Low-Rank Adaptation (SRLoRA) via importance-based fusion and reinitialization.<n> SRLoRA consistently achieves faster convergence and improved accuracy over standard LoRA.
  arXiv  Detail & Related papers  (2025-05-18T14:12:40Z)
• RaSA: Rank-Sharing Low-Rank Adaptation [67.40422142257091]
  Low-rank adaptation (LoRA) has been prominently employed for parameter-efficient fine-tuning of large language models (LLMs)<n>We introduce Rank-Sharing Low-Rank Adaptation (RaSA), an innovative extension that enhances the expressive capacity of LoRA by leveraging partial rank sharing across layers.<n>Our theoretically grounded and empirically validated approach demonstrates that RaSA not only maintains the core advantages of LoRA but also significantly boosts performance in challenging code and math tasks.
  arXiv  Detail & Related papers  (2025-03-16T17:16:36Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape [52.98187034726091]
  We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.<n>We show that Flat-LoRA improves both in-domain and out-of-domain generalization.
  arXiv  Detail & Related papers  (2024-09-22T11:24:10Z)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? [121.0693322732454]
  Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models.<n>Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning.<n>We introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of low-rank matrices.
  arXiv  Detail & Related papers  (2024-07-25T17:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.