From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning

• URL: http://arxiv.org/abs/2604.13460v1
• Date: Wed, 15 Apr 2026 04:29:00 GMT
• Title: From Order to Distribution: A Spectral Characterization of Forgetting in Continual Learning
• Abstract summary: citetevron2022catastrophic analyzes forgetting under random orderings of a fixed task collection.<n>We study an exact-fit linear regime in which tasks are sampled i.i.d. from a task distribution$$, and ask how the generating distribution itself governs forgetting.
• Score: 28.30104646642165
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: A central challenge in continual learning is forgetting, the loss of performance on previously learned tasks induced by sequential adaptation to new ones. While forgetting has been extensively studied empirically, rigorous theoretical characterizations remain limited. A notable step in this direction is \citet{evron2022catastrophic}, which analyzes forgetting under random orderings of a fixed task collection in overparameterized linear regression. We shift the perspective from order to distribution. Rather than asking how a fixed task collection behaves under random orderings, we study an exact-fit linear regime in which tasks are sampled i.i.d.\ from a task distribution~$Π$, and ask how the generating distribution itself governs forgetting. In this setting, we derive an exact operator identity for the forgetting quantity, revealing a recursive spectral structure. Building on this identity, we establish an unconditional upper bound, identify the leading asymptotic term, and, in generic nondegenerate cases, characterize the convergence rate up to constants. We further relate this rate to geometric properties of the task distribution, clarifying what drives slow or fast forgetting in this model.

Related papers

• Generalization, memorization, and overfitting for diffusion models trained in the lazy high-dimensional regime [17.718748692214508]
  We develop a generative counterpart to the theory of benign overfitting and algorithmic regularization for neural networks.<n>We derive exact risk trajectories under gradient flow training.<n>We analyze how these estimators combine along the reverse-time SDE and characterize the distribution of the resulting samples.
  arXiv  Detail & Related papers  (2026-08-25T00:54:59Z)
• Learning Permutation from Structure Without Supervision [16.16926085760151]
  Learning from structure without supervision induces a non-uniform uncertainty.<n>Existing methods control this process using a single global temperature.<n>We introduce an entropy-adaptive formulation of Gumbel-Sinkhorn that locally modulates temperature based on assignment uncertainty.
  arXiv  Detail & Related papers  (2026-05-25T08:08:47Z)
• Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training [76.12556589212666]
  We show that curriculum post-training avoids the exponential complexity bottleneck.<n>Under outcome-only reward signals, reinforcement learning finetuning achieves high accuracy with sample complexity.<n>We establish guarantees for test-time scaling, where curriculum-aware querying reduces both reward oracle calls and sampling cost from exponential to order.
  arXiv  Detail & Related papers  (2025-11-10T18:29:54Z)
• Provable In-Context Learning of Nonlinear Regression with Transformers [66.99048542127768]
  In-context learning (ICL) is the ability to perform unseen tasks using task specific prompts without updating parameters.<n>Recent research has actively explored the training dynamics behind ICL, with much of the focus on relatively simple tasks.<n>This paper investigates more complex nonlinear regression tasks, aiming to uncover how transformers acquire in-context learning capabilities.
  arXiv  Detail & Related papers  (2025-07-28T00:09:28Z)
• Rademacher learning rates for iterated random functions [0.0]
  We consider the case where the training dataset is generated by an iterated random function that is not necessarily irreducible or aperiodic.<n>Under the assumption that the governing function is contractive with respect to its first argument, we first establish a uniform convergence result for the corresponding sample error.<n>We then demonstrate the learnability of the approximate empirical risk minimization algorithm and derive its learning rate bound.
  arXiv  Detail & Related papers  (2025-06-16T19:36:13Z)
• Convergence and Implicit Bias of Gradient Descent on Continual Linear Classification [12.699007098398805]
  We study continual learning on multiple linear classification tasks by sequentially running gradient descent (GD)<n>We show the directional convergence of the trained linear classifier to the joint (offline) max-margin solution when tasks are jointly separable.<n>We also analyze the case where the tasks are no longer jointly separable and show that the model trained in a cyclic order converges to the unique minimum of the joint loss function.
  arXiv  Detail & Related papers  (2025-04-17T07:35:48Z)
• Online Inference for Quantiles by Constant Learning-Rate Stochastic Gradient Descent [4.2694059987063655]
  This paper proposes an online inference method of the gradient descent (SGD) with a constant learning rate for quantile loss functions with theoretical guarantees.<n> Numerical studies demonstrate strong finite-sample performance of our proposed quantile estimator and inference method.
  arXiv  Detail & Related papers  (2025-03-04T01:37:42Z)
• Generator Born from Classifier [66.56001246096002]
  We aim to reconstruct an image generator, without relying on any data samples. We propose a novel learning paradigm, in which the generator is trained to ensure that the convergence conditions of the network parameters are satisfied.
  arXiv  Detail & Related papers  (2023-12-05T03:41:17Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• Contrastive learning of strong-mixing continuous-time stochastic processes [53.82893653745542]
  Contrastive learning is a family of self-supervised methods where a model is trained to solve a classification task constructed from unlabeled data. We show that a properly constructed contrastive learning task can be used to estimate the transition kernel for small-to-mid-range intervals in the diffusion case.
  arXiv  Detail & Related papers  (2021-03-03T23:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.