Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

• URL: http://arxiv.org/abs/2607.26247v1
• Date: Tue, 28 Jul 2026 20:32:04 GMT
• Title: Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
• Abstract summary: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning.<n>Recent schemes initialize adapters from the downstream loss gradient.<n>We show that these seemingly distinct methods are points on a single continuum.
• Score: 0.0
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how the adapters are initialized. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature. We show that these seemingly distinct methods are points on a single continuum: a two-parameter family of preconditioned gradient initializations, which we call Unified LoRA (ULoRA), governed by a spectral whitening exponent and an Adam-like diagonal exponent. Sweeping this family under a full learning-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task-dependent and frequently lies strictly inside the family, away from the published endpoints. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine-tuning on all five GLUE tasks with RoBERTa-base and is competitive with the strongest baselines on GSM8K with LLaMA-2-7B. Our deployable, search-free variant, ULoRA-Auto, selects per-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision.

Related papers

• LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates [48.87937677659571]
  Low-rank adaptation (LoRA) is the standard way to finetune large models.<n>We introduce LoRA-TSD, an adapter that treats every LoRA step as a tangent to the fixed-rank matrix manifold.<n>We give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero.
  arXiv  Detail & Related papers  (2026-09-02T15:41:45Z)
• LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment [3.934797837010563]
  Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models.<n>LoRA-GA$2$ is an effective fine-tuning algorithm that fully leverages multi-step gradient information.
  arXiv  Detail & Related papers  (2026-08-20T08:55:05Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• Post-Optimization Adaptive Rank Allocation for LoRA [10.796887002907456]
  Post-Optimization Adaptive Rank Allocation (PARA) is a data-free compression method for Low-Rank Adaptation (LoRA)<n>We propose PARA, a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines.<n>We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across vision and language benchmarks.
  arXiv  Detail & Related papers  (2026-04-30T12:40:11Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring [9.250460219785188]
  We introduce IGU-LoRA, an adaptive-rank LoRA that computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation.<n>IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness.
  arXiv  Detail & Related papers  (2026-03-14T06:45:54Z)
• GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation [4.748720471060117]
  GRIT is a curvature-aware LoRA procedure that preserves the LoRA parameterization.<n>It matches or surpasses LoRA and QLoRA while reducing trainable parameters by 46% on average.<n>GRIT yields lower drift and a better updates-vs-retention frontier than strong PEFT-optimizer baselines.
  arXiv  Detail & Related papers  (2026-01-01T06:31:54Z)
• HRP: High-Rank Preheating for Superior LoRA Initialization [58.3319586613105]
  High-Rank Preheating (HRP) trains Low-Rank Adaptation (LoRA) with a higher preheating rank for a few steps.<n>HRP significantly enhances LoRA's effectiveness in various models and tasks.
  arXiv  Detail & Related papers  (2025-02-11T17:59:35Z)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning [73.93639228235622]
  Continual Learning with foundation models has emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks.<n>Existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks.<n>We propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal.
  arXiv  Detail & Related papers  (2025-01-22T20:00:41Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation [65.268245109828]
  We introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
  arXiv  Detail & Related papers  (2024-01-20T20:25:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.