Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation

• URL: http://arxiv.org/abs/2607.02182v1
• Date: Thu, 02 Jul 2026 13:52:12 GMT
• Title: Bayesian Sparse Low-Rank Adaptation for Large Language Model Uncertainty Estimation
• Abstract summary: Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence.<n>We propose Data-Adaptive Lower-Rank Adaptation (DALorRA), a simple and effective variational Bayesian sparse framework that shifts the paradigm of uncertainty from the dense parameter space to the lightweight rank level of low-rank adaptation (LoRA)<n>With the insight that LoRA aggregates multiple rank-one components that may provide superfluous model capacity, DALorRA imposes masking on rank dimensions, enabling regularization of model capacity during training and ensemble-like calibration during inference.
• Score: 14.828620034157316
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) exhibit remarkable reasoning capabilities, but their task-specific fine-tuning is notoriously plagued by overconfidence, severely hindering trustworthy deployment. We propose Data-Adaptive Lower-Rank Adaptation (DALorRA), a simple and effective variational Bayesian sparse framework that shifts the paradigm of uncertainty quantification from the dense parameter space to the lightweight rank level of low-rank adaptation (LoRA). With the insight that LoRA essentially aggregates multiple rank-one components that may provide superfluous model capacity, DALorRA imposes stochastic masking on rank dimensions, enabling Bayesian regularization of model capacity during training and ensemble-like calibration during inference. Extensive experiments demonstrate DALorRA's excellent calibration of LLMs without compromising reasoning accuracy.

Related papers

• BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning [40.998095775740886]
  Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces.<n>We propose BaRA, a Bayesian Adaptive Rank Allocation framework for parameter-efficient fine-tuning.<n>BaRA dynamically allocates adaptation capacity by activating a sparse, context-dependent subset of disentangled latent factors.
  arXiv  Detail & Related papers  (2026-06-28T04:08:09Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models [28.056597124418104]
  Low-Rank Adaptation (LoRA) has become the standard for fine-tuning large pre-trained models at reduced computational cost.<n>BaLoRA produces zero-shot test-time uncertainty estimates that correlate more strongly with model error than a trained ensemble of LoRA models.
  arXiv  Detail & Related papers  (2026-04-27T18:19:31Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models [5.653755499165773]
  We introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes.<n>With only approximately 0.42M additional parameters and $approx1.2times$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models up to 30B.
  arXiv  Detail & Related papers  (2026-01-28T19:54:31Z)
• C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models [19.55798373491983]
  Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs)<n>LoRA produces overconfident predictions in data-scarce few-shot settings.<n>We propose Contextual Low-Rank Adaptation (C-LoRA) as a novel uncertainty-aware and parameter efficient fine-tuning approach.
  arXiv  Detail & Related papers  (2025-05-23T11:44:02Z)
• Minimal Ranks, Maximum Confidence: Parameter-efficient Uncertainty Quantification for LoRA [9.114051379014429]
  Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large language models by decomposing weight updates into low-rank matrices.<n>LoRA lacks mechanisms for uncertainty quantification, leading to overconfident and poorly calibrated models.<n>We propose a novel parameter-efficient Bayesian LoRA via subspace inference, demonstrating that effective uncertainty quantification can be achieved in very low-dimensional parameter spaces.
  arXiv  Detail & Related papers  (2025-02-17T18:46:29Z)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning [73.93639228235622]
  Continual Learning with foundation models has emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks.<n>Existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks.<n>We propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal.
  arXiv  Detail & Related papers  (2025-01-22T20:00:41Z)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence [73.5303340531806]
  We study how Low-Rank Adaptation (LoRA) and full-finetuning change pre-trained models.<n>We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure.<n>We extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension.
  arXiv  Detail & Related papers  (2024-10-28T17:14:01Z)
• Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs [75.11449420928139]
  Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a promising solution, but there exists a gap between the practical performance of low-rank adaptations and its theoretical optimum. We propose eXtreme Gradient Boosting LoRA, a novel framework that bridges this gap by leveraging the power of ensemble learning.
  arXiv  Detail & Related papers  (2024-10-25T17:07:13Z)
• Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
  We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
  arXiv  Detail & Related papers  (2023-11-20T11:56:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.