Related papers: TaRA: Training-Aware Low-Rank Adaptation Initialization

TaRA: Training-Aware Low-Rank Adaptation Initialization

URL: http://arxiv.org/abs/2609.02639v1
Date: Wed, 02 Sep 2026 14:15:22 GMT
Title: TaRA: Training-Aware Low-Rank Adaptation Initialization
Abstract summary: Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning.<n>Training-aware Low-Rank Adaptation Initialization (TaRA) is a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix.
Score: 15.468502643386962
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.

Related papers

Normalized Low-Rank Adaptation [18.076347530341426]
We introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training.<n>NoRA consistently accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting.
arXiv Detail & Related papers (2026-08-31T16:15:36Z)
TLoRA: Task-aware Low Rank Adaptation of Large Language Models [8.568676581408239]
Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models.<n>Existing LoRA variants address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency.<n>We present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimalizes and resource allocation at the outset of training.
arXiv Detail & Related papers (2026-04-20T11:43:55Z)
LoRA-Squeeze: Simple and Effective Post-Tuning and In-Tuning Compression of LoRA Modules [10.00294036303927]
We introduce LoRA-Squeeze, a simple and efficient methodology that aims to improve standard LoRA learning.<n>Our approach posits that it is better to first learn an expressive, higher-rank solution and then compress it, rather than learning a constrained, low-rank solution directly.
arXiv Detail & Related papers (2026-02-11T16:19:58Z)
ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
arXiv Detail & Related papers (2026-02-07T10:19:36Z)
Decomposing and Composing: Towards Efficient Vision-Language Continual Learning via Rank-1 Expert Pool in a Single LoRA [50.97792275353563]
We introduce a novel framework that restructures a single Low-Rank Adaptation (LoRA) module as a decomposable Rank-1 Expert Pool.<n>Our method learns to dynamically compose a sparse, task-specific update by selecting from this expert pool, guided by the semantics of the [Guided] token.
arXiv Detail & Related papers (2026-01-30T10:54:51Z)
ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints [64.35580479051208]
In previous works, low-rank adapters (LoRA) are randomly with a fixed rank across all attachment points.<n>In this paper, we improve convergence and final performance of LoRA fine-tuning using our proposed data-driven weight initialization method.
arXiv Detail & Related papers (2025-07-09T23:52:31Z)
HRP: High-Rank Preheating for Superior LoRA Initialization [58.3319586613105]
High-Rank Preheating (HRP) trains Low-Rank Adaptation (LoRA) with a higher preheating rank for a few steps.<n>HRP significantly enhances LoRA's effectiveness in various models and tasks.
arXiv Detail & Related papers (2025-02-11T17:59:35Z)
PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation [65.268245109828]
We introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
arXiv Detail & Related papers (2024-01-20T20:25:17Z)
Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
arXiv Detail & Related papers (2023-11-20T11:56:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.