IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring

• URL: http://arxiv.org/abs/2603.13792v1
• Date: Sat, 14 Mar 2026 06:45:54 GMT
• Title: IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring
• Abstract summary: We introduce IGU-LoRA, an adaptive-rank LoRA that computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation.<n>IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness.
• Score: 9.250460219785188
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) scale to billions of parameters, full-parameter fine-tuning becomes compute- and memory-prohibitive. Parameter-efficient fine-tuning (PEFT) mitigates this issue by updating only a small set of task-specific parameters while keeping the base model frozen. Among PEFT approaches, low-rank adaptation (LoRA) is widely adopted; however, it enforces a uniform rank across layers despite substantial variation in layer importance, motivating {layerwise} rank allocation. Recent adaptive-rank variants (e.g., AdaLoRA) allocate ranks based on importance scores, yet typically rely on instantaneous gradients that capture only local sensitivity, overlooking non-local, pathwise effects within the same layer, which yields unstable and biased scores. To address this limitation, we introduce IGU-LoRA, an adaptive-rank LoRA that (i) computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation, and (ii) applies an uncertainty-aware scheme using exponential moving averages with deviation tracking to suppress noisy updates and calibrate rank selection. Theoretically, we prove an upper bound on the composite trapezoidal rule approximation error for parameter-space IG under a pathwise Hessian-Lipschitz condition, which informs the quadrature budget. Across diverse tasks and architectures, IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness. Ablations confirm the contributions of pathwise within-layer sensitivity estimates and uncertainty-aware selection to effective rank allocation. Our code is publicly available at https://github.com/withyou12/igulora.git

Related papers

• Statistical Inference for Rank Allocation in Low-Rank Adaptation [54.886931928255564]
  Low-rank adaptation (LoRA) has become a widely used parameter-efficient fine-tuning method for large language models.<n>We propose StatLoRA, a statistical inference-based rank allocation method.<n>We show that StatLoRA achieves comparable or better performance than vanilla LoRA, AdaLoRA, and I-LoRA under matched rank budgets.
  arXiv  Detail & Related papers  (2026-07-22T14:25:59Z)
• LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning [13.12609725011405]
  Low-Rank Adaptation is widely used for parameter-efficient fine-tuning.<n>We show theoretically and empirically that uniform rank allocation is fundamentally suboptimal.<n>We propose LAARA, a search-free framework that dynamically allocates ranks.
  arXiv  Detail & Related papers  (2026-07-02T19:01:59Z)
• BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning [40.998095775740886]
  Low-rank adaptation (LoRA) enables highly efficient fine-tuning by constraining task-specific updates to fixed low-rank subspaces.<n>We propose BaRA, a Bayesian Adaptive Rank Allocation framework for parameter-efficient fine-tuning.<n>BaRA dynamically allocates adaptation capacity by activating a sparse, context-dependent subset of disentangled latent factors.
  arXiv  Detail & Related papers  (2026-06-28T04:08:09Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• SMoA: Spectrum Modulation Adapter for Parameter-Efficient Fine-Tuning [57.85676271833619]
  Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning.<n>As the rank increases, more principal singular directions are preserved, which generally improves the model's performance.<n>We propose SMoA, a textbfSpectrum textbfModulation textbfAdapter that enlarges the accessible family of spectrum-aware updates.
  arXiv  Detail & Related papers  (2026-05-20T13:19:28Z)
• Not How Many, But Which: Parameter Placement in Low-Rank Adaptation [7.686602368047758]
  Given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter?<n>Under supervised fine-tuning, random and informed subsets achieve comparable performance.<n>Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy.
  arXiv  Detail & Related papers  (2026-05-12T14:46:00Z)
• Post-Optimization Adaptive Rank Allocation for LoRA [10.796887002907456]
  Post-Optimization Adaptive Rank Allocation (PARA) is a data-free compression method for Low-Rank Adaptation (LoRA)<n>We propose PARA, a data-free compression method for LoRA that integrates seamlessly into existing fine-tuning pipelines.<n>We empirically demonstrate that PARA reduces parameter count by 75-90% while preserving the predictive performance of the original, uncompressed LoRA across vision and language benchmarks.
  arXiv  Detail & Related papers  (2026-04-30T12:40:11Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factor [8.51407618718798]
  In distributed scenarios where privacy constraints necessitate Federated Learning (FL), the integration of Low-Rank Adaptation (LoRA) is often unstable.<n>This paper introduces Stabilized Federated LoRA (SFed-LoRA), a framework that theoretically characterizes the interaction between adapter rank and federated aggregation.<n>We demonstrate that SFed-LoRA prevents high-rank collapse, and achieves significantly improved stability and faster convergence.
  arXiv  Detail & Related papers  (2026-03-09T07:49:56Z)
• Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models [26.046100835887525]
  Low-Rank Adaptation (LoRA) has emerged as a promising approach to Large Language Models (LLMs) by approximating model weight updates using low-rank decomposition.<n>We propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities.<n>Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks.
  arXiv  Detail & Related papers  (2025-09-11T03:07:05Z)
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization [56.78271181959529]
  SubLoRA is a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function.<n>Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency.
  arXiv  Detail & Related papers  (2025-07-02T15:56:40Z)
• Beyond Low-Rank Tuning: Model Prior-Guided Rank Allocation for Effective Transfer in Low-Data and Large-Gap Regimes [9.4848188271008]
  Low-Rank Adaptation (LoRA) has proven effective in reducing computational costs while maintaining performance comparable to fully fine-tuned foundation models.<n>Current adaptive LoRA methods attempt to overcome this limitation by dynamically expanding or selectively allocating ranks.<n>We introduce Stable Rank-Guided Low-Rank Adaptation (SR-LoRA), a novel framework that utilizes the stable rank of pre-trained weight matrices as a natural prior for layer-wise rank allocation.
  arXiv  Detail & Related papers  (2025-06-30T23:54:23Z)
• GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning [2.7446241148152253]
  Fine-tuning large language models (LLMs) is computationally intensive because it requires updating all parameters.<n>Low-Rank Adaptation (LoRA) improves efficiency by modifying only a subset of weights but introduces a trade-off between expressivity and computational cost.<n>We propose Geometric Low-Rank Adaptation (GeLoRA), a novel framework that computes the intrinsic dimensionality of hidden state representations to adaptively select LoRA ranks.
  arXiv  Detail & Related papers  (2024-12-12T13:04:54Z)
• PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation [65.268245109828]
  We introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
  arXiv  Detail & Related papers  (2024-01-20T20:25:17Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
  We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
  arXiv  Detail & Related papers  (2023-11-20T11:56:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.