Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning

• URL: http://arxiv.org/abs/2608.11690v1
• Date: Wed, 12 Aug 2026 06:01:49 GMT
• Title: Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
• Abstract summary: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting.<n>We develop a layer-wise information-theoretic framework that separates these effects at every depth.<n>Controlled and benchmark experiments confirm the predicted memory scaling, the interior funnel, and the alignment signal's link to forgetting.
• Score: 41.371439825328544
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning must absorb new tasks without erasing old ones, and replay---mixing a small buffer of past examples into current training---is among the most effective remedies for catastrophic forgetting. Yet its generalization behavior is shaped by two coupled effects that existing analyses fold into a single hypothesis-level quantity: finite memory replaces each past distribution with an empirical proxy, and repeated reuse couples the buffer, the current data, and the final hypothesis through a shared optimization trajectory. We develop a layer-wise information-theoretic framework that separates these effects at every depth. Our main result decomposes the expected generalization gap into a replay-induced representation drift and an optimization-dependence term, the latter further resolved into stability, plasticity, interaction, and residual-coupling components. Two refinements make the framework operational. A Wasserstein relaxation of the drift term, valid under support mismatch, yields a depth-dependent drift--sensitivity trade-off whose minimizer identifies which interior layer to stabilize. An SGLD instantiation of the optimization term reduces it to a trajectory-level log-determinant budget, exposing a curvature-aware gradient-alignment statistic that serves as an online diagnostic of task-wise forgetting. Controlled and benchmark experiments confirm the predicted memory scaling, the interior funnel, and the alignment signal's link to forgetting.

Related papers

• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos [0.2864713389096699]
  We develop a mean-field theory of dropout as a perturbation of critical signal propagation at the edge of chaos.<n>We derive critical and crossover scaling laws for correlation decay and establish that smooth activations and kinked, ReLU-like activations constitute distinct classes.
  arXiv  Detail & Related papers  (2026-05-20T19:00:02Z)
• Recovering Physical Dynamics from Discrete Observations via Intrinsic Differential Consistency [12.266189649117003]
  We replace local supervision with a global structural constraint.<n>We train a time-conditioned secant velocity field whose deviation from this property serves two purposes.<n>As a training regularizer, it confines the hypothesis space to flows that compose consistently across temporal scales.
  arXiv  Detail & Related papers  (2026-05-08T20:24:32Z)
• Unbiased Gradient Estimation for Event Binning via Functional Backpropagation [64.88399635309918]
  We propose a novel framework for unbiased gradient estimation of arbitrary binning functions by synthesizing weak derivatives during backpropagation.<n>We achieve 9.4% lower EPE in self-supervised optical flow, and 5.1% lower RMS error in SLAM, demonstrating broad benefits for event-based visual perception.
  arXiv  Detail & Related papers  (2026-02-13T04:05:03Z)
• Why Some Models Resist Unlearning: A Linear Stability Perspective [7.446140380340418]
  We frame unlearning through the lens of linear coherence stability.<n>We decompose coherence along three axes: within the retain set, within the forget set, and between them.<n>To further link data properties to forgettability, we study a two layer ReLU CNN under a signal plus noise model.<n>For empirical geometry, we show that Hessian tests and CNN heatmaps align closely with the predicted boundary, mapping the stability gradient based unlearning as a function of verification, mixing, and data/model alignment.
  arXiv  Detail & Related papers  (2026-02-03T01:47:26Z)
• SPEAR++: Scaling Gradient Inversion via Sparsely-Used Dictionary Learning [48.41770886055744]
  Federated Learning has seen an increased deployment in real-world scenarios recently.<n>The introduction of the so-called gradient inversion attacks has challenged its privacy-preserving properties.<n>We introduce SPEAR, which is based on a theoretical analysis of the gradients of linear layers with ReLU activations.<n>Our new attack, SPEAR++, retains all desirable properties of SPEAR, such as robustness to DP noise and FedAvg aggregation.
  arXiv  Detail & Related papers  (2025-10-28T09:06:19Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.