Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters

• URL: http://arxiv.org/abs/2506.14530v1
• Date: Tue, 17 Jun 2025 13:55:13 GMT
• Title: Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters
• Abstract summary: Low-Rank Adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning technique for foundation models.<n>Recent work has highlighted an inherent asymmetry in the initialization of LoRA's low-rank factors.<n>This paper focuses on providing a comprehensive theoretical characterization of asymmetric LoRA with frozen random factors.
• Score: 7.687215328455751
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-Rank Adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning (PEFT) technique for foundation models. Recent work has highlighted an inherent asymmetry in the initialization of LoRA's low-rank factors, which has been present since its inception and was presumably derived experimentally. This paper focuses on providing a comprehensive theoretical characterization of asymmetric LoRA with frozen random factors. First, while existing research provides upper-bound generalization guarantees based on averages over multiple experiments, the behaviour of a single fine-tuning run with specific random factors remains an open question. We address this by investigating the concentration of the typical LoRA generalization gap around its mean. Our main upper bound reveals a sample complexity of $\tilde{\mathcal{O}}\left(\frac{\sqrt{r}}{\sqrt{N}}\right)$ with high probability for rank $r$ LoRAs trained on $N$ samples. Additionally, we also determine the fundamental limits in terms of sample efficiency, establishing a matching lower bound of $\mathcal{O}\left(\frac{1}{\sqrt{N}}\right)$. By more closely reflecting the practical scenario of a single fine-tuning run, our findings offer crucial insights into the reliability and practicality of asymmetric LoRA.

Related papers

• Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection [0.0]
  Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models.<n>The question of how to choose the LoRA rank r has no formal answer.<n>We prove an upper bound of O(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA.
  arXiv  Detail & Related papers  (2026-07-30T04:56:27Z)
• Statistical Inference for Rank Allocation in Low-Rank Adaptation [54.886931928255564]
  Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.<n>We propose StatLoRA, a statistical inference-based rank allocation method.<n>We show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and I-LoRA under matched rank budgets.
  arXiv  Detail & Related papers  (2026-07-22T14:25:59Z)
• Logistic Bandits with $\ ilde{O}(\sqrt{dT})$ Regret without Context Diversity Assumptions [1.0098114696565863]
  SupSplitLog is the first algorithm for logistic bandits that achieves $tildemathcalO(sqrtdT)$ regret without any context diversity assumption.<n>SupSplitLog strictly improves on the existing algorithms in terms of the dependence on dimension $d$ in the regret upper bound.
  arXiv  Detail & Related papers  (2026-04-24T02:21:59Z)
• Provably Adaptive Linear Approximation for the Shapley Value and Beyond [73.0940890296463]
  A fundamental and long-standing challenge is their efficient approximation.<n>We develop a linear-space algorithm that requires $P(|hatboldsymbol-boldsymbol|_2geq)leq $ for all commonly used semi-values.<n>Our algorithm allows explicit minimization of the mean square error for each specific utility function.
  arXiv  Detail & Related papers  (2026-04-09T16:38:14Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation [0.0]
  Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models.<n>We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions.
  arXiv  Detail & Related papers  (2026-02-10T22:45:47Z)
• On the Convergence Rate of LoRA Gradient Descent [22.183624306817563]
  We present a non-asymptotic convergence analysis of the textitoriginal LoRA gradient descent algorithm.<n>We prove that LoRA gradient descent converges to a stationary point at rate $O(frac1log T)$, where $T$ is the number of iterations.
  arXiv  Detail & Related papers  (2025-12-20T07:20:54Z)
• Blind Inverse Game Theory: Jointly Decoding Rewards and Rationality in Entropy-Regularized Competitive Games [0.0]
  We introduce Blind-IGT, the first statistical framework to jointly recover $theta$ and $tau$ from observed behavior.<n>We prove it achieves the optimal $mathcalO(N-1/2)$ convergence rate for joint parameter recovery.<n>We extend our framework to Markov games and demonstrate optimal convergence rates with strong empirical performance.
  arXiv  Detail & Related papers  (2025-11-07T16:27:59Z)
• How many measurements are enough? Bayesian recovery in inverse problems with general distributions [0.7366405857677226]
  We study the sample complexity of Bayesian recovery for solving inverse problems with general prior, forward operator and noise distributions.<n>As a key application, we specialize to generative priors, where $mathcalP$ is the pushforward of a latent distribution via a Deep Neural Network (DNN)
  arXiv  Detail & Related papers  (2025-05-15T18:11:54Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models [7.108651381160281]
  Low-Rank Adaptation (LoRA) strategy balances efficiency and performance in fine-tuning large models. We propose textbfCoRA: leveraging shared knowledge to optimize LoRA training by substituting its matrix $B$ with a common subspace from large models. Our experiments show that the first approach achieves the same efficacy as the original LoRA fine-tuning while being more efficient than halving parameters.
  arXiv  Detail & Related papers  (2024-08-31T12:48:27Z)
• Random pairing MLE for estimation of item parameters in Rasch model [22.32547146723177]
  Rasch model is widely used in psychometrics to model the relationship between individuals' latent traits and their binary responses. We introduce a new likelihood-based estimator that faithfully estimates the item parameters in the Rasch model. We provide empirical evidence of the efficacy of the two new estimators using both simulated and real data.
  arXiv  Detail & Related papers  (2024-06-20T04:32:34Z)
• Sparse PCA with Oracle Property [115.72363972222622]
  We propose a family of estimators based on the semidefinite relaxation of sparse PCA with novel regularizations. We prove that, another estimator within the family achieves a sharper statistical rate of convergence than the standard semidefinite relaxation of sparse PCA.
  arXiv  Detail & Related papers  (2023-12-28T02:52:54Z)
• Optimal Multi-Distribution Learning [88.3008613028333]
  Multi-distribution learning seeks to learn a shared model that minimizes the worst-case risk across $k$ distinct data distributions. We propose a novel algorithm that yields an varepsilon-optimal randomized hypothesis with a sample complexity on the order of (d+k)/varepsilon2.
  arXiv  Detail & Related papers  (2023-12-08T16:06:29Z)
• The Expressive Power of Low-Rank Adaptation [11.371811534310078]
  Low-Rank Adaptation, a parameter-efficient fine-tuning method, has emerged as a prevalent technique for fine-tuning pre-trained models. This paper takes the first step to bridge the gap by theoretically analyzing the expressive power of LoRA. For Transformer networks, we show any model can be adapted to a target model of the same size with rank-$(fractextembedding size2)$ LoRA.
  arXiv  Detail & Related papers  (2023-10-26T16:08:33Z)
• LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models [104.23434818428062]
  We focus on the scenario where quantization and LoRA fine-tuning are applied together on a pre-trained model. We propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework. Experiments show that our method is highly effective and outperforms existing quantization methods.
  arXiv  Detail & Related papers  (2023-10-12T18:34:08Z)
• Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation [107.54516740713969]
  We study human-in-the-loop reinforcement learning (RL) with trajectory preferences. Instead of receiving a numeric reward at each step, the agent only receives preferences over trajectory pairs from a human overseer. We propose the first optimistic model-based algorithm for PbRL with general function approximation.
  arXiv  Detail & Related papers  (2022-05-23T09:03:24Z)
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity [67.02490430380415]
  We show that model-based MARL achieves a sample complexity of $tilde O(|S||B|(gamma)-3epsilon-2)$ for finding the Nash equilibrium (NE) value up to some $epsilon$ error. We also show that such a sample bound is minimax-optimal (up to logarithmic factors) if the algorithm is reward-agnostic, where the algorithm queries state transition samples without reward knowledge.
  arXiv  Detail & Related papers  (2020-07-15T03:25:24Z)
• Sharp Statistical Guarantees for Adversarially Robust Gaussian Classification [54.22421582955454]
  We provide the first result of the optimal minimax guarantees for the excess risk for adversarially robust classification. Results are stated in terms of the Adversarial Signal-to-Noise Ratio (AdvSNR), which generalizes a similar notion for standard linear classification to the adversarial setting.
  arXiv  Detail & Related papers  (2020-06-29T21:06:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.