The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning

• URL: http://arxiv.org/abs/2607.23711v1
• Date: Sun, 26 Jul 2026 15:17:41 GMT
• Title: The Intruder Threshold: A Spectral Law for LoRA Fine-Tuning
• Abstract summary: LoRA fine-tuning can create intruder dimensions that drive catastrophic forgetting.<n>No theory has predicted, layer by layer on measured spectra, when they appear.<n>Full fine-tuning disperses its update far below the threshold of every layer.<n>Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting.
• Score: 3.860898721527665
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LoRA fine-tuning can create intruder dimensions: new leading singular vectors of the updated weight matrix $W+BA$ that are nearly orthogonal to all pretrained singular vectors and that drive catastrophic forgetting. Since their discovery, no theory has predicted, layer by layer on measured spectra, when they appear. We derive a per-layer critical update strength $s^\ast=\barθ/(γσ_1(BA))$, computed from the measured spectrum of $W$ alone through the rectangular spiked-deformation transform, together with an exact secular-equation characterization of the updated spectrum, with no fitted parameters. In a pre-specified study spanning four dense Transformer families, a state-space model, a mixture-of-experts model, and an encoder-decoder (18 adapters, 9{,}840 layer scans), the law localizes the empirical threshold within a factor of two on $82\%$ of layers, separates intruder-bearing from intruder-free layers at deployment with a mean AUC of $0.89$, holds unchanged on six third-party adapters, and predicts where WikiText-2 perplexity begins to degrade; a combination of the two pre-specified edge evaluations reaches $98\%$ and is confirmed out-of-bag on the external adapters ($0.997$). Full fine-tuning disperses its update far below the threshold of every layer, which resolves the asymmetry between LoRA and full fine-tuning. Norm-matched interventions confirm that threshold-crossing layers, rather than update magnitude, carry the forgetting, and a spike-budget rule derived from the thresholds, requiring one SVD and no validation sweeps, reduces forgetting by $62\%$ on the most fragile model at no task cost.

Related papers

• Two-Stage Deformable-Convolutional Inverse Design of Nanophotonic Absorbers from Optical Spectra [2.4149105714758545]
  Data-driven inverse design enables efficient generation of nanophotonic structures with prescribed optical responses.<n>This work presents a two-stage deformable-convolutional framework for reconstructing metal--insulator--metal resonator from 80-dimensional absorption spectra.
  arXiv  Detail & Related papers  (2026-08-12T09:47:06Z)
• Saturation Makes Quantization Error Additive: A Coverage Model with a Certificate [0.0]
  Mixed-precision quantization must decide which parts of a model to keep at higher precision.<n>We show that the loss from quantizing a set of layers can be reconstructed from per-layer or pairwise sensitivities measured in isolation.<n>We propose the coverage model $f(S)=cbigl (1-prod_iin S (1-a_i)bigr)$, which reproduces the measured variance profile of $f$ to within a few percent from its $L$ fitted break-rates.
  arXiv  Detail & Related papers  (2026-07-14T02:08:33Z)
• Layer-Isolated Evaluation: Gating the Deterministic Scaffold of a Production LLM Agent with a No-LLM, Regression-Locked Test Harness [45.148328075418156]
  A deployed ordering agent is decomposed into a fixed taxonomy of layers.<n>The pure suite runs in CI on every change against a locked per-slice baseline.<n>We validate by controlled regression injection, degrading one layer at a time across seven non-safety layers.
  arXiv  Detail & Related papers  (2026-06-10T05:55:13Z)
• Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training [45.88028371034407]
  We present a parameter estimator of the trace of the diagonal blocks of the Hessian matrix of the empirical risk of a neural network.<n>The procedure combines the Hutchinson trace estimator with a single Hessian-vector product over the whole parameter vector.<n>We show that correctness under weight sharing requires the layer-wise Hessian to be assembled before the second differentiation.
  arXiv  Detail & Related papers  (2026-05-25T10:24:32Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• Weight Decay Regimes in Grokking Transformers: Cheap Online Diagnostics [0.0]
  Transformers trained on modular arithmetic exhibit sharp transitions between memorization, generalization, and collapse.<n>We show that weight decay acts as a scalar empirical control parameter for these regimes.<n>We introduce online diagnostics, mean pairwise attentionhead cosine similarity and entropy standard deviation, that track training dynamics.
  arXiv  Detail & Related papers  (2026-05-19T19:48:40Z)
• Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters [48.3175117923623]
  Most gradient clipping rules treat all parameters as heavy vectors and ignore the matrix structure of modern vectors.<n>We show empirically that data outliers often amplify only a small number of leading singular values.<n>Motivated by this phenomenon, we propose clipping, which stabilizes by singular values.
  arXiv  Detail & Related papers  (2026-05-12T09:24:59Z)
• Spectral Edge Dynamics of Training Trajectories: Signal--Noise Geometry Across Scales [0.0]
  We show that transformer training trajectories evolve within only a few coherent directions.<n>In companion work, the same spectral geometry provides early-warning signals of grokking.
  arXiv  Detail & Related papers  (2026-03-14T04:46:05Z)
• Beyond Accuracy: A Unified Random Matrix Theory Diagnostic Framework for Crash Classification Models [6.908972852063454]
  We introduce a diagnostic framework grounded in Random Matrix Theory (RMT) and Heavy-Tailed Self-Regularization (HTSR)<n>We evaluate nine model families on two Iowa DOT crash classification tasks (173,512 and 371,062 records respectively)<n>We find that the power-law exponent $$ provides a structural quality signal: well-regularized models consistently yield $$ within $[2, 4]$ (mean $2.87 pm 0.34$)<n>We propose an $$-based early stopping criterion and a spectral model selection protocol, and validate both against cross-validated F
  arXiv  Detail & Related papers  (2026-02-23T05:42:54Z)
• Phase Transition for Budgeted Multi-Agent Synergy [41.486076708302456]
  Multi-agent systems can improve reliability, yet under a fixed inference budget they often help, saturate, or even collapse.<n>We develop a minimal and calibratable theory that predicts these regimes from three binding constraints of modern agent stacks.
  arXiv  Detail & Related papers  (2026-01-24T05:32:50Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• FedSVD: Adaptive Orthogonalization for Private Federated Learning with LoRA [68.44043212834204]
  Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)<n>Low-Rank Adaptation (LoRA) is widely used for efficient fine-tuning of language models in learning (FL)
  arXiv  Detail & Related papers  (2025-05-19T07:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.