AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation

• URL: http://arxiv.org/abs/2605.08734v1
• Date: Sat, 09 May 2026 06:37:44 GMT
• Title: AdaPreLoRA: Adafactor Preconditioned Low-Rank Adaptation
• Abstract summary: Low-Rank Adaptation (LoRA) re-models a weight update as a product of two low-rank factors.<n>AdaPreLoRA fills this gap by adopting the Adaecker preconditioner $H_t$ on $W$ and selecting from the resulting factor-space solution family.
• Score: 1.9996763865817948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-Rank Adaptation (LoRA) reparameterizes a weight update as a product of two low-rank factors, but the Jacobian $J_{G}$ of the generator mapping the factors to the weight matrix is rank-deficient, so the factor-space preconditioner $J_{G}^* {F}_t J_{G}$ induced by any ${W}$-space preconditioner ${F}_t$ is singular, and consequently the standard chain rule cannot be uniquely inverted to map a preconditioned ${W}$-space direction back to a factor-space update. We cast existing LoRA optimizers in a unified framework parameterized by two choices: (i) which invertible surrogate for $J_{G}^* {F}_t J_{G}$ to use, and (ii) which ${F}_t$ on ${W}$ to use. Existing methods occupy four families along these axes: factor-space adaptive updates, block-diagonal surrogates for $J_{G}^* J_{G}$, Frobenius-residual pseudoinverse methods, and Riemannian manifold constraint. Within this design space, a gradient-statistics-aware ${F}_t$ paired with a closed-form factor-space solve at ${O}((m+n)r)$ memory remains underexplored. We propose \textbf{AdaPreLoRA}, which fills this gap by adopting the Adafactor diagonal Kronecker preconditioner ${H}_t$ on ${W}$ and selecting from the resulting factor-space solution family the element minimizing an ${H}_t$-weighted imbalance between the two factor contributions; by construction, the resulting factor update is the closest LoRA approximation to the preconditioned ${W}$-space direction under the ${H}_t$-weighted norm. Across GPT-2 (E2E), Mistral-7B and Qwen2-7B (GLUE, ARC, GSM8K), and diffusion-model personalization, AdaPreLoRA is competitive with or improves over a representative set of LoRA optimizers while keeping peak GPU memory at the LoRA optimizer level.

Related papers

• On the Convergence of Stochastic Low-Rank Adaptation [46.97973925956649]
  Low-rank adaptation (LoRA) optimize $J(B,A)=mathcal L(W_mathrmbase+sBA)$ over two $B.<n>We propose LoRA-NSGDM, which finds an $$-stationary point with $mathcalO(-8)$ oracle complexity.
  arXiv  Detail & Related papers  (2026-07-24T04:41:26Z)
• Bias-Controlled Primal-Dual Natural Actor-Critic: Optimal Rates for Constrained Multi-Objective Average-Reward RL [41.64418624570687]
  We propose a primal-dual Natural Actor-Critic algorithm for average-reward MDPs.<n>We show that the algorithm achieves optimal global convergence and constraint-violation rates of $ tildeO (1/sqrtT) $.
  arXiv  Detail & Related papers  (2026-06-23T17:56:10Z)
• AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning [17.98348342954707]
  This paper shows that the bottleneck is a measurement-topology problem rather than a need for an external subspace.<n> AR1-ZO pairs alternating rank-$1$ atom queries with topology-aware scaling $=r$, restoring rank-invariant active signal.<n> Experiments on OPT and Qwen3 models validate the signal mechanism and show that AR1-ZO makes high-rank LoRA effective.
  arXiv  Detail & Related papers  (2026-05-19T12:39:30Z)
• Group Representational Position Encoding [66.33026480082025]
  We present GRAPE, a unified framework for positional encoding based on group actions.<n>Two families of mechanisms: (i) multiplicative rotations (Multiplicative GRAPE) in $mathrmSO(d)$ and (ii) additive logit biases (Additive GRAPE) arising from unipotent actions in the general linear group $mathrmGL$.
  arXiv  Detail & Related papers  (2025-12-08T18:39:13Z)
• Evolution Strategies at the Hyperscale [57.75314521465674]
  We introduce EGGROLL, an evolution strategies (ES) algorithm designed to scale backprop-free optimization to large population sizes.<n>ES is a set of powerful blackbox optimisation methods that can handle non-differentiable or noisy objectives.<n>EGGROLL overcomes these bottlenecks by generating random matrices $Ain mathbbRmtimes r, Bin mathbbRntimes r$ with $rll min(m,n)$ to form a low-rank matrix perturbation $A Btop$
  arXiv  Detail & Related papers  (2025-11-20T18:56:05Z)
• Parameter-free Algorithms for the Stochastically Extended Adversarial Model [59.81852138768642]
  Existing approaches for the Extended Adversarial (SEA) model require prior knowledge of problem-specific parameters, such as the diameter of the domain $D$ and the Lipschitz constant of the loss functions $G$.<n>We develop parameter-free methods by leveraging the Optimistic Online Newton Step (OONS) algorithm to eliminate the need for these parameters.
  arXiv  Detail & Related papers  (2025-10-06T10:53:37Z)
• BoRA: Towards More Expressive Low-Rank Adaptation with Block Diversity [23.25105718896569]
  Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method widely used in large language models.<n>We propose Block Diversified Low-Rank Adaptation (BoRA), which improves the rank of LoRA weights with a small number of additional parameters.
  arXiv  Detail & Related papers  (2025-08-09T11:58:39Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)
• Efficient Over-parameterized Matrix Sensing from Noisy Measurements via Alternating Preconditioned Gradient Descent [17.422662003404586]
  We propose an alternating preconditioned gradient descent (APGD) algorithm for noisy matrix sensing problems.<n>APGD consistently achieves the fastest convergence and the lowest computation time compared to the existing alternatives.
  arXiv  Detail & Related papers  (2025-02-01T15:44:39Z)
• CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models [7.108651381160281]
  Low-Rank Adaptation (LoRA) strategy balances efficiency and performance in fine-tuning large models. We propose textbfCoRA: leveraging shared knowledge to optimize LoRA training by substituting its matrix $B$ with a common subspace from large models. Our experiments show that the first approach achieves the same efficacy as the original LoRA fine-tuning while being more efficient than halving parameters.
  arXiv  Detail & Related papers  (2024-08-31T12:48:27Z)
• Efficient Frameworks for Generalized Low-Rank Matrix Bandit Problems [61.85150061213987]
  We study the generalized low-rank matrix bandit problem, proposed in citelu2021low under the Generalized Linear Model (GLM) framework. To overcome the computational infeasibility and theoretical restrain of existing algorithms, we first propose the G-ESTT framework. We show that G-ESTT can achieve the $tildeO(sqrt(d_1+d_2)3/2Mr3/2T)$ bound of regret while G-ESTS can achineve the $tildeO
  arXiv  Detail & Related papers  (2024-01-14T14:14:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.