Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory

• URL: http://arxiv.org/abs/2605.10658v1
• Date: Mon, 11 May 2026 14:41:08 GMT
• Title: Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory
• Abstract summary: Continual learning requires new-task adaptation without damaging previously acquired capabilities.<n>Recent forward-pass and zeroth-order (ZO) results show that low-query adaptation may retain better than first-order descent.<n>We give a local randomized gradient-shaping analysis: finite differences expose a raw shape that is mean-aligned with FO.<n>For norm-matched ZO, the expected shaped retention curvature obeys an exact identity that preserves the isotropic retention floor while contracting only the anisotropic component.<n>Projecting this identity onto the incoming gradient yields the observable FO--ZO quadratic forgetting gap.
• Score: 23.912005398880293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning requires new-task adaptation without damaging previously acquired capabilities. Recent forward-pass and zeroth-order (ZO) results show that low-query adaptation may retain better than first-order (FO) descent, but the usual view of ZO as noisy FO estimation does not explain why. We give a local randomized gradient-shaping analysis: finite differences expose a raw shape that is mean-aligned with FO, while the norm-matched comparator fixes the expected squared adaptation norm. Under this controlled comparison, forgetting depends on how the adaptation shape exposes retention curvature. For norm-matched ZO, the expected shaped retention curvature obeys an exact identity that preserves the isotropic retention floor while contracting only the anisotropic component. Projecting this identity onto the incoming gradient yields the observable FO--ZO quadratic forgetting gap: ZO improves mean forgetting precisely when the FO direction has above-average retention curvature, by a query-dependent fraction of that curvature excess. A practical finite-query accounting separates the mean mechanism from one-batch sampling and smoothing perturbations. As an algorithmic transfer, RISE applies the calibrated ZO shape to exact FO gradients inside parameter blocks. Its target is a stability--plasticity tradeoff: randomized shaping may reduce the retention exposure paid by FO, exact gradients remove finite-smoothing bias from finite-difference ZO, and blockwise sampling supplies many local shaping directions after one gradient computation. The blockwise analysis separates mean-step damage from centered random exposure, showing how block-diagonal curvature, cross-block coupling, and local shaping diagnostics specify where this exact-gradient transfer is most likely to be visible.

Related papers

• Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization [7.792020418343022]
  Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning.<n>We show that the two methods are locally consistent under parameter-space perturbations.<n>We also show that divergence-based regularization is defined through transformations of the input.
  arXiv  Detail & Related papers  (2026-09-08T19:01:05Z)
• Scale-Consistent Posterior Dynamics for Diffusion Inverse Problems [16.685352002333225]
  Posterior sampling with a pretrained diffusion prior is governed by a conditional score whose likelihood component is generally intractable.<n>We show that a noise-conditioned path whose targets approach the clean posterior produces a continuous surrogate SDE.<n>Experiments on FFHQ and ImageNet with 100 score evaluations demonstrate competitive reconstruction fidelity for superresolution and deblurring.
  arXiv  Detail & Related papers  (2026-08-15T09:38:41Z)
• Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models [49.46165710676499]
  We find that semantic convergence is observed from the first alignment stage.<n>We find that convergence can be revealed and amplified, but not introduced by the SFT data.<n> Taken together, our results suggest that semantic convergence may arise naturally from the objectives underlying LM training.
  arXiv  Detail & Related papers  (2026-08-11T20:47:06Z)
• Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data [3.7437735326653883]
  This paper studies the rate-controlled middle case for full-batch linear classification on separable data.<n>For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice.
  arXiv  Detail & Related papers  (2026-07-07T08:55:35Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• Conditional Diffusion Under Linear Constraints: Langevin Mixing and Information-Theoretic Guarantees [10.73598638822051]
  We study zero-shot conditional sampling with pretrained diffusion models for linear inverse problems.<n>We prove that the error from replacing this score by the unconditional tangent score is upper bounded by a dimension-free conditional mutual information between observed and unobserved components.
  arXiv  Detail & Related papers  (2026-05-06T19:19:54Z)
• From Video-to-PDE: Data-Driven Discovery of Nonlinear Dye Plume Dynamics [0.0]
  Inferring continuum models directly from video is hampered by two facts: the recorded field is uncalibrated image intensity rather than a physical state.<n>We develop a video-to-PDE pipeline that converts grayscale recordings of an ink plume into a normalised scalar field.<n>We show that uncalibrated visual data can yield compact, predictive and structurally interpretable continuum models when discovery, calibration and uncertainty are treated as distinct stages.
  arXiv  Detail & Related papers  (2026-05-06T06:23:59Z)
• Conservation Law Breaking at the Edge of Stability: A Spectral Theory of Non-Convex Neural Network Optimization [0.0]
  We show that gradient descent reliably finds good solutions in non-dimensional network optimization.<n>We identify two regimes separated by a width-dependent dynamical transition.<n>All predictions are validated across 23 experiments.
  arXiv  Detail & Related papers  (2026-04-08T10:41:24Z)
• Distribution-Free Confidence Ellipsoids for Ridge Regression with PAC Bounds [0.8594140167290097]
  This paper introduces an extension to the SPS EOA algorithm to ridge regression, and derives probably approximately correct (PAC) upper bounds for the resulting region sizes.
  arXiv  Detail & Related papers  (2026-01-19T22:47:39Z)
• Learning-Based Shrinking Disturbance-Invariant Tubes for State- and Input-Dependent Uncertainty [1.3750624267664155]
  We develop a learning-based framework for constructing shrinking disturbance-invariant tubes under state- and input-dependent uncertainty.<n>We certify safety via a lifted, isotone (order-preserving) fixed-point map.
  arXiv  Detail & Related papers  (2026-01-16T16:47:04Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Unlearning-based Neural Interpretations [51.99182464831169]
  We show that current baselines defined using static functions are biased, fragile and manipulable.<n>We propose UNI to compute an (un)learnable, debiased and adaptive baseline by perturbing the input towards an unlearning direction of steepest ascent.
  arXiv  Detail & Related papers  (2024-10-10T16:02:39Z)
• Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent [43.097493761380186]
  gradient algorithms are an efficient method of approximately solving linear systems. We show that gradient descent produces accurate predictions, even in cases where it does not converge quickly to the optimum. Experimentally, gradient descent achieves state-of-the-art performance on sufficiently large-scale or ill-conditioned regression tasks.
  arXiv  Detail & Related papers  (2023-06-20T15:07:37Z)
• Differentiable Annealed Importance Sampling and the Perils of Gradient Noise [68.44523807580438]
  Annealed importance sampling (AIS) and related algorithms are highly effective tools for marginal likelihood estimation. Differentiability is a desirable property as it would admit the possibility of optimizing marginal likelihood as an objective. We propose a differentiable algorithm by abandoning Metropolis-Hastings steps, which further unlocks mini-batch computation.
  arXiv  Detail & Related papers  (2021-07-21T17:10:14Z)
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective [68.61254575987013]
  Deep learning models are increasingly popular in many machine learning applications where the training data may contain sensitive information. Many learning systems now incorporate differential privacy by training their models with (differentially) private SGD. A key step in each private SGD update is gradient clipping that shrinks the gradient of an individual example whenever its L2 norm exceeds some threshold.
  arXiv  Detail & Related papers  (2020-06-27T19:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.