Related papers: On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference

URL: http://arxiv.org/abs/2605.29580v1
Date: Thu, 28 May 2026 08:24:45 GMT
Title: On the Construction and Implications of Low-Loss Valleys in LoRA-based Bayesian Inference
Abstract summary: We introduce LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space with two variants.<n>We prove pathwise continuity and Lipschitz regularity of the loss along the curve and empirically show, across reasoning and classification benchmarks with Qwen2.5 7B, that linear perturbations encounter loss barriers.<n>Combined with flat-minima perturbations and a Jensen-Shannon divergence regularizer, LoRA-Curve yields measurably higher mutual information of the predictive distribution without sacrificing performance.
Score: 12.367241512406656
License: http://creativecommons.org/licenses/by/4.0/
Abstract: While parameter-efficient fine-tuning methods like low-rank adaptation (LoRA) are standard for large language models, principled estimation of epistemic uncertainty remains challenging. Recent results in the LoRA regime suggest that discrete multi-mode approaches such as deep ensembles offer little benefit over single-mode methods. This contradicts broader observations in deep learning, where ensembling independent optima typically improves generalization, and linking these modes through continuous low-loss valleys further enhances Bayesian model averaging (BMA). Whether such structure exists in the LoRA space and whether it yields functional diversity missed by local or discrete methods has not been studied. We introduce LoRA-Curve, a segmented Bézier curve parameterization in the LoRA space, with two variants: a free configuration that jointly optimizes all control points, and an anchored configuration that connects independently fine-tuned LoRA optima. We prove pathwise continuity and Lipschitz regularity of the loss along the curve and empirically show, across reasoning and classification benchmarks with Qwen2.5 7B, that linear interpolation encounters loss barriers, while our anchored multi-segment curves connect independent optima through continuous low-loss valleys. Combined with flat-minima perturbations and a Jensen-Shannon divergence regularizer, LoRA-Curve yields measurably higher mutual information of the predictive distribution without sacrificing performance, and links continuous parameter-space traversal to functional diversity.

Related papers

FAN-LoRA: A Fourier-Adaptive Nonlinear Low-Rank Adaptor for Medical Foundation Model Domain Adaptation [11.04143971119602]
We present FAN-LoRA, a novel frequency-decoupled fine-tuning architecture.<n>We show that FAN-LoRA consistently outperforms state-of-the-art PEFT baselines.<n>Compared to the strongest competitors, our method achieves consistent improvements in average Dice scores and notable reductions in boundary errors.
arXiv Detail & Related papers (2026-08-27T02:08:57Z)
ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation [4.391214332238358]
Decision-support in digital agriculture requires accurate predictions and well-calibrated uncertainty estimates.<n>We propose ST-LoRA, a parameter-efficient ensemble framework that builds diverse ensemble members from a single training trajectory.<n> ST-LoRA matches or exceeds full-rank ensembles in segmentation accuracy and calibration across both datasets and architectures.
arXiv Detail & Related papers (2026-08-02T22:49:28Z)
Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
arXiv Detail & Related papers (2026-05-21T16:45:31Z)
OmniISR: A Unified Framework for Centralized and Federated Learning via Intermediate Supervision and Regularization [58.03221830946145]
We propose OmniISR, a unified framework that fuses pure CL, pure FL, and hybrid CL-FL training modes.<n>We show that OmniISR consistently improves model performance in both centralized and federated paradigms.
arXiv Detail & Related papers (2026-05-19T04:13:27Z)
DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
arXiv Detail & Related papers (2026-05-17T07:14:44Z)
Scalable Variational Bayesian Fine-Tuning of LLMs via Orthogonalized Low-Rank Adapters [13.718993776070434]
Uncertainty quantification (UQ) is of utmost importance to self-assess the reliability of large language models (LLMs)<n>We build on the Bayesian last layer (BLL) model, where the LLM-based deterministic feature extractor is followed by random last layer parameters for uncertainty reasoning.<n>We leverage the variational (V) inference framework to put forth a scalable Bayesian fine-tuning approach.
arXiv Detail & Related papers (2026-04-03T18:42:04Z)
Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
arXiv Detail & Related papers (2026-03-31T14:46:39Z)
ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
arXiv Detail & Related papers (2026-02-07T10:19:36Z)
Adaptive Federated LoRA in Heterogeneous Wireless Networks with Independent Sampling [15.218221234361922]
Federated LoRA has emerged as a technique for efficiently fine-tuning large language models on distributed devices.<n>In this paper, we propose independent federated convergence wall-clock time of fine-tuning under both system and data heterogeneity.<n>Experiments demonstrate that our approach reduces wall-clock time compared to state-of-the-art methods across various models and datasets.
arXiv Detail & Related papers (2025-05-29T15:31:37Z)
AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mapping [28.37735374308455]
AuroRA: (I) not only matches or surpasses full fine-tuning performance with only 6.18% 25% of LoRA's parameters but also (II) outperforms state-of-the-art PEFT methods by up to 10.88% in both NLP and CV tasks.
arXiv Detail & Related papers (2025-05-24T15:16:27Z)
Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation [58.288682735160585]
Low-Rank Adaptation (LoRA) is a popular technique for finetuning models. LoRA often under performs when compared to full- parameter fine-tuning. We present a framework that rigorously analyzes the adaptation rates of LoRA methods.
arXiv Detail & Related papers (2024-10-10T18:51:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.