Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

• URL: http://arxiv.org/abs/2607.27680v1
• Date: Thu, 30 Jul 2026 04:56:27 GMT
• Title: Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection
• Abstract summary: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models.<n>The question of how to choose the LoRA rank r has no formal answer.<n>We prove an upper bound of O(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood. Existing generalization results provide upper bounds of the form O~(sqrt(rd/n)) or O~(rd/n), but a matching lower bound is missing, and the question of how to choose the LoRA rank r has no formal answer. Both gaps are closed here. A local Rademacher argument establishes an upper bound of O~(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA, whenever the target adaptation has rank at most r. A matching minimax lower bound of Omega(rd/n) is then proved via a Fano-type packing of the rank-r subspace of R^{d x d}; the bound applies to any estimator whose output lies in the rank-r LoRA class. Combining the two yields a rank-selection dichotomy. For the constrained empirical risk minimizer, the optimal rank equals the intrinsic rank r*, and over-ranking strictly hurts. For adaptive estimators of the nuclear-norm-then-truncate type, over-ranking is harmless and the rate saturates at Theta~(r* d / n) regardless of r. Taken together, the three results characterize the statistical complexity of LoRA fine-tuning within the well-specified locally quadratic regime, and identify the empirically observed over-parameterization penalty as a property of unregularized empirical risk minimization rather than of the LoRA class itself. Predictions of the theory are verified on a synthetic trace-regression benchmark and on real LoRA fine-tuning across three (model, task) configurations covering DistilBERT and RoBERTa on SST-2 and MRPC. All configurations exhibit the predicted U-shape in validation loss, with two showing statistically significant loss inflation at large ranks (paired permutation p = 0.016).

Related papers

• Statistical Inference for Rank Allocation in Low-Rank Adaptation [54.886931928255564]
  Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.<n>We propose StatLoRA, a statistical inference-based rank allocation method.<n>We show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and I-LoRA under matched rank budgets.
  arXiv  Detail & Related papers  (2026-07-22T14:25:59Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence [30.96331195818476]
  Low-Rank Adaptation (LoRA) is the most widely adopted method for fine-tuning large language models.<n>We introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold.<n>BaLoRA converges faster than standard LoRA and achieves superior performance across a range of fine-tuning tasks.
  arXiv  Detail & Related papers  (2026-05-29T16:08:28Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• LoRA vs. Full Fine-Tuning: A Theoretical Perspective [29.827767726672523]
  Low-Rank Adaptation (LoRA) is an efficient fine-tuning method that reduces memory and computation costs.<n>We study LoRA in a simple linear regression setting and compare its excess risk with that of full fine-tuning.<n>Our theory predicts that LoRA can outperform full fine-tuning when the difference between the pretraining and the downstream tasks is effectively low-rank.
  arXiv  Detail & Related papers  (2026-05-18T18:40:24Z)
• Rethinking the Rank Threshold for LoRA Fine-Tuning [4.221888521641282]
  A recent landscape analysis of LoRA fine-tuning in the neural tangent kernel regime establishes a sufficient condition $r(r+1)/2 > KN$ on the LoRA rank $r$ for the absence of spurious local minima under squared-error loss.<n>We give three results that together reduce the prescribed rank to $r = 1$ for binary classification in this regime.
  arXiv  Detail & Related papers  (2026-05-05T13:09:46Z)
• Post-Optimization Adaptive Rank Allocation for LoRA [10.796887002907456]
  Post-Optimization Adaptive Rank Allocation (PARA) is a data-free compression method for Low-Rank Adaptation (LoRA)<n>We propose PARA, a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines.<n>We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across vision and language benchmarks.
  arXiv  Detail & Related papers  (2026-04-30T12:40:11Z)
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization [56.78271181959529]
  SubLoRA is a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function.<n>Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency.
  arXiv  Detail & Related papers  (2025-07-02T15:56:40Z)
• Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters [7.687215328455751]
  Low-Rank Adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning technique for foundation models.<n>Recent work has highlighted an inherent asymmetry in the initialization of LoRA's low-rank factors.<n>This paper focuses on providing a comprehensive theoretical characterization of asymmetric LoRA with frozen random factors.
  arXiv  Detail & Related papers  (2025-06-17T13:55:13Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• LoRA Training in the NTK Regime has No Spurious Local Minima [46.46792977614938]
  Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models. We theoretically analyze LoRA fine-tuning in the neural tangent kernel regime with $N$ data points.
  arXiv  Detail & Related papers  (2024-02-19T06:22:09Z)
• Fast and Robust Rank Aggregation against Model Misspecification [105.54181634234266]
  In rank aggregation (RA), a collection of preferences from different users are summarized into a total order under the assumption of homogeneity of users. Model misspecification in RA arises since the homogeneity assumption fails to be satisfied in the complex real-world situation. We propose CoarsenRank, which possesses robustness against model misspecification.
  arXiv  Detail & Related papers  (2019-05-29T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.