AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery

• URL: http://arxiv.org/abs/2605.10741v1
• Date: Mon, 11 May 2026 15:44:13 GMT
• Title: AdaPaD: Adaptive Parallel Deflation for PEFT with Self-Correcting Rank Discovery
• Abstract summary: We present AdaPaD (Adaptive Parallel Deflation), which trains all rank-1 components simultaneously.<n>We prove that every component's error decays exponentially after a warm-up period.<n>AdaPaD is competitive with adaptive-rank LoRA baselines on GLUE with DeBERTaV3-base at matched parameter budgets.
• Score: 10.176501817419371
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning large language models with LoRA requires choosing a rank r before training starts. Existing approaches either extract rank-1 components sequentially, freezing each component's error permanently into every subsequent residual, or optimize the full low-rank factorization jointly with guarantees that describe only the joint update, not individual rank-1 directions. We present AdaPaD (Adaptive Parallel Deflation), which trains all rank-1 components simultaneously: each worker refines its component against a deflation target built from the latest estimates of all predecessors, and as those estimates improve, the targets improve too. We call this property self-correction: deflation errors converge to zero over rounds rather than persisting as fixed residuals. On top of this backbone, AdaPaD adds advance learning (private pre-training before activation) and per-module dynamic rank discovery (importance-based growth until a shared budget is exhausted), making the rank distribution an output rather than an input. We prove that every component's error decays exponentially after a warm-up period, with a generalization bound that splits into a vanishing algorithmic term and an irreducible statistical floor. Empirically, AdaPaD is competitive with adaptive-rank LoRA baselines on GLUE with DeBERTaV3-base at matched parameter budgets, and competitive with fixed-rank LoRA on Qwen3-0.6B SQuAD/SQuAD v2 while deploying an adapter that is on average 30.7% smaller.

Related papers

• PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR [0.0]
  Reinforcement learning with verifiable rewards spends most of its compute generating groups of long reasoning trajectories.<n>Recent allocators reduce this cost by assigning budgets to prompts, rollouts, or tokens according to a pointwise notion of difficulty or utility.<n>We identify a statistical mismatch: the unclipped leave-one-out group-relative score gradient is not a sum of independent point contributions, but a second-order U-statistic over pairs of rollouts.
  arXiv  Detail & Related papers  (2026-08-11T19:24:59Z)
• ARC: Augmented-Rank Conformalization for Changepoint Localization --- Finite-Sample Validity and Distribution-Robust Efficiency [7.648414136980082]
  ARC is a family of scores depending on the data only through within-segment ranks.<n>Every ARC score inherits finite-sample coverage for every frozen weight configuration.<n>The entire ARC confidence set is almost surely invariant under strictly increasing marginal transforms.
  arXiv  Detail & Related papers  (2026-08-09T02:44:08Z)
• UniRank: Unified List-wise Reranking via Confidence-Ordered Denoising [33.296591996616975]
  List-wise reranking arranges a request-specific pool of candidate items into an ordered slate that maximizes user satisfaction.<n>We present UniRank, a unified list-wise reranking framework whose inference time variants recover AR and NAR rerankers as special cases.<n>Experiments on Amazon Books, MovieLens-1M, and an industrial short video dataset show that UniRank consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-11T13:12:23Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• A Self-Attentive Meta-Optimizer with Group-Adaptive Learning Rates and Weight Decay [0.42970700836450487]
  We propose a new modulation that integrates a self-attention mechanism to dynamically modulate per-group learning rates and weight decay.<n>To train the attention module, we introduce a meta-learning objective that combines gradient alignment, loss decrease, and gap.<n>A key novel contribution is the extension of homoscedastic uncertainty weighting (HUW) with task-specific priorities that directly scale the regularization terms.
  arXiv  Detail & Related papers  (2026-04-10T02:56:01Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• CAIRO: Decoupling Order from Scale in Regression [13.755937210012883]
  We propose a framework that decouples regression into two distinct stages.<n>In the first stage, we learn a scoring function by minimizing a scale-invariant ranking loss.<n>In the second, we recover the target scale via isotonic regression.
  arXiv  Detail & Related papers  (2026-02-16T03:50:05Z)
• Ensemble Threshold Calibration for Stable Sensitivity Control [0.0]
  We present an end-to-end framework that achieves exact recall with sub-percent variance over tens of millions of geometry pairs.<n>Our approach consistently hits a recall target within a small error, decreases redundant verifications relative to other calibrations, and runs end-to-end on a single TPU v3 core.
  arXiv  Detail & Related papers  (2025-10-02T15:22:28Z)
• Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization [56.78271181959529]
  SubLoRA is a rank determination method for Low-Rank Adaptation (LoRA) based on submodular function.<n>Our method combines solid theoretical foundations, second-order accuracy, and practical computational efficiency.
  arXiv  Detail & Related papers  (2025-07-02T15:56:40Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks [52.46420522934253]
  We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks.<n>The method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble.
  arXiv  Detail & Related papers  (2024-05-23T11:10:32Z)
• Learning by Minimizing the Sum of Ranked Range [58.24935359348289]
  We introduce the sum of ranked range (SoRR) as a general approach to form learning objectives. A ranked range is a consecutive sequence of sorted values of a set of real numbers. We explore two applications in machine learning of the minimization of the SoRR framework, namely the AoRR aggregate loss for binary classification and the TKML individual loss for multi-label/multi-class classification.
  arXiv  Detail & Related papers  (2020-10-05T01:58:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.