Not How Many, But Which: Parameter Placement in Low-Rank Adaptation

• URL: http://arxiv.org/abs/2605.12207v1
• Date: Tue, 12 May 2026 14:46:00 GMT
• Title: Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
• Abstract summary: Given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter?<n>Under supervised fine-tuning, random and informed subsets achieve comparable performance.<n>Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy.
• Score: 7.686602368047758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the \textit{parameter placement problem}: given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter? Under supervised fine-tuning, random and informed subsets achieve comparable performance. Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy. This regime dependence traces to gradient structure: SFT gradients are low-rank and directionally stable, so any subset accumulates coherent updates; GRPO gradients are high-rank and near-orthogonal across steps, so only elements with consistently signed gradients retain the learning signal. Our scoring procedure identifies these critical parameters in under 10 seconds at less than 0.5% of training cost. Selected parameters concentrate on residual-stream-writing projections (V, O, Down), stable across model families and scales (1.5B - 8B).

Related papers

• Adaptively Incorporating Directional Hints into Zeroth-Order Optimization [49.940317286021696]
  We develop an algorithm that achieves zeroth-order optimization of non- functions with the aid directional hints.<n>We validate the method, demonstrating sustained progress on non- control tasks.
  arXiv  Detail & Related papers  (2026-09-08T05:42:02Z)
• SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning [70.71795707610961]
  method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
  arXiv  Detail & Related papers  (2026-08-29T21:34:47Z)
• SubZero+: Efficient Zeroth-Order LLM Fine-Tuning via Large Learning Rates [25.09609527559127]
  SubZero+ is an improved SubZero framework that improves stability in three complementary ways.<n>We show that SubZero+ consistently outperforms prior ZO baselines, enlarges the stable learning-rate range, and narrows the gap to first-order methods with minimal extra memory overhead.
  arXiv  Detail & Related papers  (2026-08-16T10:13:25Z)
• Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps [4.422349568747053]
  We study the phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining.<n>We give a provable account of which case obtains robustness, and it turns on two properties together: the structure of the gradient noise and whether the structure of the gradient noise normalizes its update.
  arXiv  Detail & Related papers  (2026-07-14T05:21:38Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Robust stochastic first order methods in heavy-tailed noise via medoid mini-batch gradient sampling [1.439518478021091]
  We consider a first order optimization framework where, at each iteration, $K$ independent identically distributed (i.i.d.) data point samples are drawn.<n>In this paper, we take an alternative approach and propose a novel first order gradient algorithm dubbed Robust Descent GradientMini.<n> Experimental results indicate that R-SGD-Mini and its clipped variant consistently perform favorably compared to Median-of-Means based methods.
  arXiv  Detail & Related papers  (2026-05-08T12:01:25Z)
• IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring [9.250460219785188]
  We introduce IGU-LoRA, an adaptive-rank LoRA that computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation.<n>IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness.
  arXiv  Detail & Related papers  (2026-03-14T06:45:54Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning [55.03441672267886]
  We propose GradAlign, a gradient-aligned data selection method for reinforcement learning.<n>We evaluate GradAlign across three data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus.
  arXiv  Detail & Related papers  (2026-02-25T01:54:50Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• Rough Path Signatures: Learning Neural RDEs for Portfolio Optimization [0.0]
  We introduce a deep BSDE/2BSDE solver that couples truncated log-signatures with a neural rough differential equation backbone.<n>The architecture aligns objective analysis with sequence-to-path learning: a CVaR-tilted terminal targets left-tail risk, while an optional second-order (2BSDE) head supplies curvature estimates for risk-sensitive control.<n>At d=200 it achieves CVaR(0.99)=9.80% versus 12.00-13.10% for strong baselines, attains the lowest HJB residual (0.011), and yields the lowest RMSEs for Z and Gamma.
  arXiv  Detail & Related papers  (2025-10-12T18:02:12Z)
• PLUMAGE: Probabilistic Low rank Unbiased Min Variance Gradient Estimator for Efficient Large Model Training [21.695928776150808]
  Accelerator memory and networking constraints have emerged as dominant bottlenecks when training large language models.<n>We propose PLUMAGE: Probabilistic Low rank Unbiased Minimum vAriance Gradient Estor.<n>We empirically demonstrate that PLUMAGE shrinks the full rank optimization's gap over the pre training evaluation loss by 33% on average across models and the average training loss across the GLUE benchmark by 28% within a similar computational and memory footprint as GaloRE.
  arXiv  Detail & Related papers  (2025-05-23T19:17:55Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Scaling Forward Gradient With Local Losses [117.22685584919756]
  Forward learning is a biologically plausible alternative to backprop for learning deep neural networks. We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
  arXiv  Detail & Related papers  (2022-10-07T03:52:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.