Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation

• URL: http://arxiv.org/abs/2606.18024v1
• Date: Tue, 16 Jun 2026 15:03:37 GMT
• Title: Catastrophic Forgetting is Low-Rank: A Function-Space Theory for Continual Adaptation
• Abstract summary: New-task training induces old-task prediction drift through the cross-task kernel.<n>In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision.
• Score: 5.220809937383775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Catastrophic forgetting in continual adaptation is usually studied through parameter drift, replay, or distillation, but these views do not identify which output-space directions are vulnerable. We give a function-space account in the NTK regime: new-task training induces old-task prediction drift through the cross-task kernel, yielding a closed-form predictor for the forgetting vector before any new-task gradient step. In frozen-backbone linear-head PEFT-CL, where the model is linear in the trainable parameters, the predictor is exact up to numerical precision; for nonlinear adapters/full fine-tuning, it is a local NTK approximation. The same expression reveals that forgetting concentrates in a small number of old-task NTK eigenmodes and under frozen linear heads gives a Kronecker scaling rule for the vulnerable rank. These results clarify the relation to prior NTK-overlap theory, explain why parameter-space regularizers can miss output-space interference, and motivate a targeted spectral regularizer.

Related papers

• Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks [3.980892205755342]
  We present an orientation-resolved quadratic master formula that predicts the random-slice residual from both the curvature spectrum and the reference-to-solution displacement profile.<n>We then introduce Random Mapping Networks (RaMaN), which instantiate the predicted latent dimension using structured Hadamard or seed-regenerated Gaussian maps.<n>RaMaN closely tracks measured transition locations and outperforms orientation-agnostic approximations when displacement direction matters.
  arXiv  Detail & Related papers  (2026-08-12T21:09:54Z)
• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• The Global Empirical NTK: Self-Referential Bias and Dimensionality of Gradient Descent Learning [13.950010831924823]
  We study the structure of the Global Empirical Neural Tangent Kernel (NTK) for a range of models.<n>We show that the NTK is structurally bottlenecked, constraining its effective rank and giving rise to a self-referential bias.<n>Overall, we show that the NTK possesses tractable structure that explains GD bias toward task solutions.
  arXiv  Detail & Related papers  (2026-05-09T07:22:29Z)
• Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation [10.547646302449682]
  Finetuning pretrained models occurs in a low-dimensional subspace of the full parameter space.<n>Are stable directions irrelevant to downstream tasks, or do they already encode task-relevant structure that requires no further adjustment?<n>We show that the leading singular vectors of pretrained weight matrices remain highly stable under finetuning and are shared across unrelated downstream tasks.
  arXiv  Detail & Related papers  (2026-05-08T06:12:43Z)
• Curvature-Guided LoRA: Steering in the pretrained NTK subspace [60.35296431630704]
  We introduce the prediction alignment problem, aiming to match the predictor obtained via PEFT to that of full fine-tuning at the level of outputs.<n>We show that this objective naturally leads to a curvature-aware, second-order formulation, where optimal low-rank updates correspond to a Newton-like, curvature-whitened gradient.<n>Based on this insight, we propose Curvature-Guided LoRA (CG-LoRA), which selects and scales adaptation directions using local curvature information.
  arXiv  Detail & Related papers  (2026-03-31T14:46:39Z)
• The Curvature Rate λ: A Scalar Measure of Input-Space Sharpness in Neural Networks [0.0]
  Curvature influences generalization, robustness, and how reliably neural networks respond to small input perturbations.<n>We introduce a scalar curvature measure defined directly in input space: the curvature rate lambda.<n>lambda tracks the emergence of high-frequency structure in the decision boundary.
  arXiv  Detail & Related papers  (2025-11-03T10:46:03Z)
• Nonlocal Neural Tangent Kernels via Parameter-Space Interactions [0.0]
  The Neural Tangent Kernel (NTK) has provided insights into the training dynamics of neural networks under gradient flow.<n>We propose a Nonlocal Neural Tangent Kernel (NNTK) that replaces the local gradient with a nonlocal interaction-based approximation in parameter space.
  arXiv  Detail & Related papers  (2025-09-15T21:23:47Z)
• On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigm [47.55215041326702]
  We discover an intriguing linear phenomenon in models that are from a common pretrained checkpoint and finetuned on different tasks, termed as Cross-Task Linearity (CTL) We show that if we linearly interpolate the weights of two finetuned models, the features in the weight-interpolated model are often approximately equal to the linearities of features in two finetuned models at each layer. We conjecture that in the pretraining-finetuning paradigm, neural networks approximately function as linear maps, mapping from the parameter space to the feature space.
  arXiv  Detail & Related papers  (2024-02-06T03:28:36Z)
• Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels [78.6096486885658]
  We introduce lower bounds to the linearized Laplace approximation of the marginal likelihood. These bounds are amenable togradient-based optimization and allow to trade off estimation accuracy against computational complexity.
  arXiv  Detail & Related papers  (2023-06-06T19:02:57Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Disentangling Generative Factors of Physical Fields Using Variational Autoencoders [0.0]
  This work explores the use of variational autoencoders (VAEs) for non-linear dimension reduction. A disentangled decomposition is interpretable and can be transferred to a variety of tasks including generative modeling.
  arXiv  Detail & Related papers  (2021-09-15T16:02:43Z)
• Spectral Tensor Train Parameterization of Deep Learning Layers [136.4761580842396]
  We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. We show the effects of neural network compression in the classification setting and both compression and improved stability training in the generative adversarial training setting.
  arXiv  Detail & Related papers  (2021-03-07T00:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.