LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates

• URL: http://arxiv.org/abs/2609.02734v1
• Date: Wed, 02 Sep 2026 15:41:45 GMT
• Title: LoRA-TSD: Tangent-Space Spectral Descent for LoRA via Muon-Style Updates
• Abstract summary: Low-rank adaptation (LoRA) is the standard way to finetune large models.<n>We introduce LoRA-TSD, an adapter that treats every LoRA step as a tangent to the fixed-rank matrix manifold.<n>We give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero.
• Score: 48.87937677659571
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-rank adaptation (LoRA) is the standard way to fine-tune large models, yet when its two factors are trained independently, the update ignores the geometry of the low-rank weight change it induces. We introduce LoRA-TSD, an optimizer that treats every LoRA step as a tangent vector of the fixed-rank matrix manifold and takes the spectral-norm steepest-descent step of Muon inside that tangent space, mapping the result back to the factors through a retraction native to the LoRA parametrization. The step avoids expensive operations on full weight matrices, and its retraction is up to $2.8\times$ cheaper than the truncated-SVD retraction used by prior manifold methods. We prove that the Frobenius-norm version of our surrogate recovers LoRA-Pro, and we identify the tangent-projected gradient, the Riemannian gradient of the manifold, as the stationarity measure natural to LoRA training and computable from the factor gradients alone. Under this measure we give the first global convergence guarantees for both LoRA-Pro and LoRA-TSD, with rates that drive the factor-gradient norms to zero. Across six commonsense and natural-language-inference benchmarks with Llama-3.2-1B, Llama-3.1-8B and Qwen3-32B, LoRA-TSD outperforms every competing LoRA optimizer and stays robust to the adapter rank. Code is available at https://github.com/brain-lab-research/LoRA-TSD.

Related papers

• LoRA-GA$^2$: Low Rank Adaptation with Multi-step Gradient Adaptive Alignment [3.934797837010563]
  Low-Rank Adaptation (LoRA) is a prominent fine-tuning method for large models.<n>LoRA-GA$2$ is an effective fine-tuning algorithm that fully leverages multi-step gradient information.
  arXiv  Detail & Related papers  (2026-08-20T08:55:05Z)
• SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling [17.873016968298483]
  Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.<n>We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that re parameterizes a rank-rtot update as a sum of K static experts.
  arXiv  Detail & Related papers  (2026-06-27T03:25:36Z)
• Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection [13.686308852232857]
  We propose Echo-LoRA, a cross-layer representation injection method for parameter-efficient fine-tuning.<n>During training, Echo-LoRA collects boundary hidden states from deeper source layers, aggregates them into a sample-level echo representation, and uses lightweight projection networks to inject the resulting signal into shallow LoRA or DoRA modules.<n>On eight commonsense reasoning benchmarks, Echo-LoRA exceeds the reported LoRA baseline on average across LLaMA-7B, LLaMA2-7B, and LLaMA3-8B.
  arXiv  Detail & Related papers  (2026-05-05T08:26:11Z)
• Beyond SGD, Without SVD: Proximal Subspace Iteration LoRA with Diagonal Fractional K-FAC [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>In this work, we address the gap between training with full steps with low-rank projections (SVDLoRA) and LoRA fine-tuning.<n>We propose LoRSum, a memory-efficient subroutine that closes this gap for gradient descent.
  arXiv  Detail & Related papers  (2026-02-18T13:41:41Z)
• GRIT -- Geometry-Aware PEFT with K-FACPreconditioning, Fisher-Guided Reprojection, andDynamic Rank Adaptation [4.748720471060117]
  GRIT is a curvature-aware LoRA procedure that preserves the LoRA parameterization.<n>It matches or surpasses LoRA and QLoRA while reducing trainable parameters by 46% on average.<n>GRIT yields lower drift and a better updates-vs-retention frontier than strong PEFT-optimizer baselines.
  arXiv  Detail & Related papers  (2026-01-01T06:31:54Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• BeamLoRA: Beam-Constraint Low-Rank Adaptation [51.52097743781401]
  Low-Rank Adaptation (LoRA) has been widely adopted as one of the most effective parameter-efficient fine-tuning methods.<n>We propose BeamLoRA, which conceptualizes each LoRA module as a beam where each rank naturally corresponds to a potential sub-solution.
  arXiv  Detail & Related papers  (2025-02-19T10:33:22Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• LoRA-Pro: Are Low-Rank Adapters Properly Optimized? [121.0693322732454]
  Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models.<n>Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning.<n>We introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of low-rank matrices.
  arXiv  Detail & Related papers  (2024-07-25T17:57:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.