Behaviorally Effective LoRA Writes Are Sparse and Structured

• URL: http://arxiv.org/abs/2609.01374v1
• Date: Tue, 01 Sep 2026 15:09:43 GMT
• Title: Behaviorally Effective LoRA Writes Are Sparse and Structured
• Abstract summary: Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained write actually carry behavior.<n>We show that behaviorally effective LoRA writes are sparse, structured, and far more concentrated than the raw low-rank parameterization suggests.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-rank adaptation fixes the rank of the update, but it does not identify which parts of a trained write actually carry behavior. We study that question directly and show that behaviorally effective LoRA writes are sparse, structured, and far more concentrated than the raw low-rank parameterization suggests. We use Learned-Basis LoRA, a learned-basis continuation recipe, to expose that structure. The recipe warms up an unconstrained adapter, converts its learned write columns into a module-wise orthonormal basis, freezes that basis, and continues training inside the constrained parameterization. Across 14 exact switches from unconstrained to constrained form, held-out accuracy is unchanged at the conversion step and reconstructed write matrices differ by at most 0.25% relative Frobenius error. Same-state continuation then shows that the same trained checkpoint develops differently under different write subspaces, establishing write geometry as a causal state variable. A no-retraining projection test shows that useful write signal stays inside the learned write space and largely disappears from random or frozen-activation PCA controls. The concentration pattern is strong at both local and global scales. Across GSM8K, MathQA, and AQuA, per-module top-k continuation reaches its optimum at k in {2, 4} in all twelve seed-level cases we test. A stricter global ranking test shows that learned top-16 and top-32 subsets outperform matched random subsets, especially on GSM8K/Qwen and MathQA/Qwen. Single-direction ablations further reveal a sparse set of late q_proj, o_proj, and down_proj components with outsized behavioral impact.

Related papers

• Structuring Semantic Embeddings for Principle Evaluation: A Prototype-Guided Contrastive Learning Approach [58.15427952222424]
  General-purpose text embeddings are widely deployed for such tasks, but broad semantic similarity can place semantically similar yet task-distinct examples.<n>We introduce Prototype-Guided Contrastive Learning (PGCL), a prototype-guided geometric regularization module built on top of frozen text embeddings.
  arXiv  Detail & Related papers  (2026-08-15T13:19:25Z)
• ARC: Augmented-Rank Conformalization for Changepoint Localization --- Finite-Sample Validity and Distribution-Robust Efficiency [7.648414136980082]
  ARC is a family of scores depending on the data only through within-segment ranks.<n>Every ARC score inherits finite-sample coverage for every frozen weight configuration.<n>The entire ARC confidence set is almost surely invariant under strictly increasing marginal transforms.
  arXiv  Detail & Related papers  (2026-08-09T02:44:08Z)
• DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models [26.283978881046107]
  Reinforcement learning with verifiable rewards improves the reasoning capabilities of large language models.<n>On-policy self-distillation mitigates signal sparsity by querying a privileged teacher at student-visited prefixes.<n>Standard OPSD still underexploits the temporal structure of the rollout.<n>DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate.
  arXiv  Detail & Related papers  (2026-08-06T16:29:24Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Instruction Tuning Changes How Upstream State Conditions Late Readout: A Cross-Patching Diagnostic [9.8812664524155]
  We ask how earlier and the late stack cooperate to turn differences into next-token margins.<n>We cross each model's earlier-layer state with each model's late stack.<n>Forced-token scoring shows that the local token choice can change later exact-answer success.
  arXiv  Detail & Related papers  (2026-05-08T05:47:10Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring [9.250460219785188]
  We introduce IGU-LoRA, an adaptive-rank LoRA that computes within-layer Integrated Gradients (IG) sensitivities and aggregates them into a layer-level score for rank allocation.<n>IGU-LoRA consistently outperforms strong PEFT baselines at matched parameter budgets, improving downstream accuracy and robustness.
  arXiv  Detail & Related papers  (2026-03-14T06:45:54Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• Consistent-Teacher: Towards Reducing Inconsistent Pseudo-targets in Semi-supervised Object Detection [28.40887130075552]
  Pseudo-targets undermine the training of an accurate detector. It injects noise into the student's training, leading to severe overfitting problems. We propose a systematic solution, termed ConsistentTeacher, to reduce the inconsistency.
  arXiv  Detail & Related papers  (2022-09-04T10:21:02Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.