CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation

• URL: http://arxiv.org/abs/2607.02576v1
• Date: Tue, 30 Jun 2026 20:00:47 GMT
• Title: CORA: Per-Slice Coherent Orthogonal Rotation for SVD-based Low-Rank Adaptation
• Abstract summary: We show that stable finetuning follows a coherent SVD rotation.<n> CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA on commonsense reasoning and code generation.
• Score: 6.432245831737748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Parameter-Efficient Fine-Tuning (PEFT) commonly adapts pretrained weights through low-rank updates, and recent methods further exploit the singular value decomposition (SVD) of the base weight for initialization or subspace selection. However, these methods do not explicitly preserve the coupled geometry between the pretrained left and right singular bases. Motivated by recent minimum-perturbation theory, which shows that stable finetuning follows a coherent SVD rotation in which a single orthogonal $Q$ acts on both the left singular basis $U_0$ and the right singular basis $V_0$, we prove a per-slice analogue: each row slice of $W_0$ can be adapted by a shared orthogonal rotation $Q_i$ on its left basis $U_i$ and right basis $V_i$ together with a diagonal spectrum shift. We implement this form as CORA (Coherent Orthogonal Rotation Adaptation), which applies per-slice orthogonal rotations and a per-layer diagonal scale to the rank-$r$ SVD truncation of $W_0$. CORA uses $\tfrac{1}{2}m(r{-}1)$ trainable parameters per linear layer, about $4{\times}$ fewer than LoRA at the same rank. CORA outperforms LoRA, DoRA, PiSSA, and MiLoRA on commonsense reasoning and code generation while using about $8{\times}$ fewer parameters.

Related papers

• On the Convergence of Stochastic Low-Rank Adaptation [46.97973925956649]
  Low-rank adaptation (LoRA) optimize $J(B,A)=mathcal L(W_mathrmbase+sBA)$ over two $B.<n>We propose LoRA-NSGDM, which finds an $$-stationary point with $mathcalO(-8)$ oracle complexity.
  arXiv  Detail & Related papers  (2026-07-24T04:41:26Z)
• Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation [0.0]
  Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models.<n>We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions.
  arXiv  Detail & Related papers  (2026-02-10T22:45:47Z)
• Group Representational Position Encoding [66.33026480082025]
  We present GRAPE, a unified framework for positional encoding based on group actions.<n>Two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $mathrmSO(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $mathrmGL$.
  arXiv  Detail & Related papers  (2025-12-08T18:39:13Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold [51.93627542334909]
  Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models.<n>We propose a geometry-aware extension of LoRA that uses a three-factor decomposition $U!SVtop$.
  arXiv  Detail & Related papers  (2025-10-02T11:59:13Z)
• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs [26.212332132619736]
  We propose an asymmetric multi-LoRA design with multiple $A$ matrices and a single shared $B$ in multi-task fine-tuning.<n>Our methods achieve more balanced performance across tasks with comparable or superior average accuracy relative to existing multi-LoRA approaches.
  arXiv  Detail & Related papers  (2025-09-29T19:16:14Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Models [10.827800772359844]
  We study the computational limits of Low-Rank Adaptation (LoRA) for finetuning transformer-based models using fine-grained complexity theory.<n>Our key observation is that the existence of low-rank decompositions within the gradient computation of LoRA adaptation leads to possible algorithmic speedup.
  arXiv  Detail & Related papers  (2024-06-05T10:44:08Z)
• A Single Linear Layer Yields Task-Adapted Low-Rank Matrices [4.695004706877747]
  Low-Rank Adaptation (LoRA) is a widely used Efficient Fine-Tuning (PEFT) method that updates an initial weight matrix $W_0$ with a delta matrix $Delta W$. We show that CondLoRA maintains a performance on par with LoRA, despite the fact that the trainable parameters of CondLoRA are fewer than those of LoRA.
  arXiv  Detail & Related papers  (2024-03-22T04:38:42Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.