BayesLoRA: Task-Specific Uncertainty in Low-Rank Adapters

• URL: http://arxiv.org/abs/2506.22809v1
• Date: Sat, 28 Jun 2025 08:22:02 GMT
• Title: BayesLoRA: Task-Specific Uncertainty in Low-Rank Adapters
• Authors: Cooper Doyle,
• Abstract summary: BayesLoRA provides guardrails tailored to downstream, enabling agents to introspect and modulate behavior under uncertainty.<n>We demonstrate mathematically and empirically that LoRA adapters exhibit amplified variance outside fine-tuning distributions, yielding reliable confidence estimates for agentic decision-making.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose BayesLoRA, a task-specific uncertainty quantification framework that integrates MC-Dropout into Low-Rank Adapters (LoRA). Unlike general-purpose transformer uncertainty methods, BayesLoRA provides guardrails tailored to downstream workflows, enabling agents to introspect and modulate behavior under uncertainty. We demonstrate mathematically and empirically that LoRA adapters exhibit amplified variance outside fine-tuning distributions, yielding reliable confidence estimates for agentic decision-making.

Related papers

• Towards Minimal Fine-Tuning of VLMs [59.01498204407219]
  Image-LoRA is a lightweight parameter efficient fine-tuning recipe for transformer-based vision-language models.<n>Image-LoRA applies low-rank adaptation only to the value path of attention layers within the visual-token span.<n>It matches or closely approaches standard LoRA accuracy while using fewer trainable parameters and lower adapter-only training FLOPs.
  arXiv  Detail & Related papers  (2025-12-22T10:02:10Z)
• LoFA: Learning to Predict Personalized Priors for Fast Adaptation of Visual Generative Models [50.46815266062554]
  Low-Rank Adaptation (LoRA) remains impractical due to their demand for task-specific data and lengthy optimization.<n>We propose LoFA, a general framework that efficiently predicts personalized priors for fast model adaptation.<n>Our method consistently predicts high-quality personalized priors within seconds, across multiple tasks and user prompts, even outperforming conventional LoRA that requires hours of processing.
  arXiv  Detail & Related papers  (2025-12-09T16:39:31Z)
• DiffLoRA: Differential Low-Rank Adapters for Large Language Models [59.58987161199141]
  We introduce DiffLoRA, a parameter-efficient adaptation of the differential attention mechanism, with low-rank adapters on both positive and negative attention terms.<n>We evaluate DiffLoRA across a broad range of NLP tasks, including general benchmarks, many-shot in-context learning, RAG, and long-context tests.
  arXiv  Detail & Related papers  (2025-07-31T14:24:59Z)
• Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes [9.4848188271008]
  Low-Rank Adaptation (LoRA) has proven effective in reducing computational costs while maintaining performance comparable to fully fine-tuned foundation models.<n>Current adaptive LoRA methods attempt to overcome this limitation by dynamically expanding or selectively allocating ranks.<n>We introduce Stable Rank-Guided Low-Rank Adaptation (SR-LoRA), a novel framework that utilizes the stable rank of pre-trained weight matrices as a natural prior for layer-wise rank allocation.
  arXiv  Detail & Related papers  (2025-06-30T23:54:23Z)
• PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models [13.074320303580361]
  Low-Rank Adaptation (LoRA) is a widely used finetuning method for large models.<n>We introduce PLoP (Precise LoRA Placement), a lightweight method that allows automatic identification of adapters.<n>We demonstrate that PLoP consistently outperforms, and in the worst case competes, with commonly used placement strategies.
  arXiv  Detail & Related papers  (2025-06-25T17:25:02Z)
• FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts [47.35092228595656]
  FLoE is a novel PEFT framework that introduces two key innovations: (i) a Fisher information-guided importance scoring mechanism to dynamically identify task-critical transformer layers for MoE-based low-rank adaptation, enabling sparse adapter deployment; and (ii) a Bayesian optimization-driven rank allocator that automatically determines optimal LoRA ranks on specific datasets without exhaustive grid search.<n>Experiments across diverse LLMs and benchmarks reveal that FLoE achieves impressive efficiency-accuracy trade-offs, making FLoE particularly advantageous in resource-constrained environments that necessitate rapid adaptation.
  arXiv  Detail & Related papers  (2025-05-31T10:27:08Z)
• C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models [16.30304620457414]
  Low-Rank Adaptation (LoRA) offers a cost-effective solution for fine-tuning large language models (LLMs)<n>LoRA produces overconfident predictions in data-scarce few-shot settings.<n>We propose Contextual Low-Rank Adaptation (textbfC-LoRA) as a novel uncertainty-aware and parameter efficient fine-tuning approach.
  arXiv  Detail & Related papers  (2025-05-23T11:44:02Z)
• Interpretable Risk Mitigation in LLM Agent Systems [0.0]
  We explore agent behaviour in a toy, game-theoretic environment based on a variation of the Iterated Prisoner's Dilemma.<n>We introduce a strategy-modification method-independent of both the game and the prompt-by steering the residual stream with interpretable features extracted from a sparse autoencoder latent space.
  arXiv  Detail & Related papers  (2025-05-15T19:22:11Z)
• SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation [41.91948079316541]
  Recent studies have raised concerns that LoRA fine-tuning could potentially compromise the safety alignment in large language models.<n>In this paper, we propose Safety-alignment preserved Low-Rank Adaptation (SaLoRA)<n>Unlike previous LoRA methods and their variants, SaLoRA enables targeted modifications to LLMs without disrupting their original alignments.<n>Our experiments show that SaLoRA outperforms various adapters-based approaches across various evaluation metrics in different fine-tuning tasks.
  arXiv  Detail & Related papers  (2025-01-03T11:34:28Z)
• Unlocking Tuning-Free Few-Shot Adaptability in Visual Foundation Models by Recycling Pre-Tuned LoRAs [76.40876036912537]
  Large Language Models (LLMs) demonstrate strong few-shot adaptability without requiring fine-tuning.<n>Current Visual Foundation Models (VFMs) require explicit fine-tuning with sufficient tuning data.<n>We propose a framework, LoRA Recycle, that distills a meta-LoRA from diverse pre-tuned LoRAs with a meta-learning objective.
  arXiv  Detail & Related papers  (2024-12-03T07:25:30Z)
• RILQ: Rank-Insensitive LoRA-based Quantization Error Compensation for Boosting 2-bit Large Language Model Accuracy [5.767260383077013]
  Low-rank adaptation (LoRA) has become the dominant method for parameter-efficient LLM fine-tuning.<n>LoRA-based quantization error compensation (LQEC) emerging as a powerful tool for recovering accuracy in compressed LLMs.<n>We propose RILQ (Rank-Insensitive LoRA-based Quantization Error Compensation) to understand fundamental limitation and boost 2-bit LLM accuracy.
  arXiv  Detail & Related papers  (2024-12-02T05:09:56Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• LoRA vs Full Fine-tuning: An Illusion of Equivalence [76.11938177294178]
  We study how Low-Rank Adaptation (LoRA) and full-finetuning change pre-trained models.<n>We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure.<n>We extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension.
  arXiv  Detail & Related papers  (2024-10-28T17:14:01Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
  Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
  arXiv  Detail & Related papers  (2024-10-10T18:51:53Z)
• Low-Rank Adaptation Secretly Imitates Differentially Private SGD [5.359060261460183]
  We show theoretically that low-rank adaptation is equivalent to fine-tuning adapters with noisy batch gradients.<n>We also quantify the variance of the injected noise as a decreasing function of adaptation rank.<n>Low-rank adaptation provides robustness to membership inference attacks w.r.t the fine-tuning data.
  arXiv  Detail & Related papers  (2024-09-26T04:56:49Z)
• Rapid Switching and Multi-Adapter Fusion via Sparse High Rank Adapters [16.160749645651567]
  We propose Sparse High Rank Adapters (SHiRA) that directly finetune 1-2% of the base model weights while leaving others unchanged. This high sparsity incurs no inference overhead, enables rapid switching directly in the fused mode, and significantly reduces concept-loss during multi-adapter fusion.
  arXiv  Detail & Related papers  (2024-07-22T22:46:36Z)
• DoRA: Weight-Decomposed Low-Rank Adaptation [57.68678247436207]
  We introduce a novel weight decomposition analysis to investigate the inherent differences between FT and LoRA. Aiming to resemble the learning capacity of FT from the findings, we propose Weight-Decomposed Low-Rank Adaptation (DoRA) DoRA decomposes the pre-trained weight into two components, magnitude and direction, for fine-tuning.
  arXiv  Detail & Related papers  (2024-02-14T17:59:34Z)
• Meta-Learning Adversarial Bandits [49.094361442409785]
  We study online learning with bandit feedback across multiple tasks, with the goal of improving average performance across tasks if they are similar according to some natural task-similarity measure. As the first to target the adversarial setting, we design a meta-algorithm that setting-specific guarantees for two important cases: multi-armed bandits (MAB) and bandit optimization (BLO) Our guarantees rely on proving that unregularized follow-the-leader combined with multiplicative weights is enough to online learn a non-smooth and non-B sequence.
  arXiv  Detail & Related papers  (2022-05-27T17:40:32Z)
• Learning Calibrated Uncertainties for Domain Shift: A Distributionally Robust Learning Approach [150.8920602230832]
  We propose a framework for learning calibrated uncertainties under domain shifts. In particular, the density ratio estimation reflects the closeness of a target (test) sample to the source (training) distribution. We show that our proposed method generates calibrated uncertainties that benefit downstream tasks.
  arXiv  Detail & Related papers  (2020-10-08T02:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.