Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

• URL: http://arxiv.org/abs/2608.19762v1
• Date: Thu, 20 Aug 2026 08:07:01 GMT
• Title: Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
• Abstract summary: A minibatch can influence training beyond update at which it is observed because AdamW stores past information in its states.<n>We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence.
• Score: 8.328108675535562
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.

Related papers

• Same Loss, Same Noise, Opposite Schedules: Noise Structure and Optimizer Normalization Jointly Determine Whether Learning-Rate Cooldown Helps [4.422349568747053]
  We study the phase of a warmup-stable-decay (WSD) learning-rate schedule, now a default in large-model pretraining.<n>We give a provable account of which case obtains robustness, and it turns on two properties together: the structure of the gradient noise and whether the structure of the gradient noise normalizes its update.
  arXiv  Detail & Related papers  (2026-07-14T05:21:38Z)
• From Atoms to Entropy: Optimal Noise Allocation for Diffusion Training in the Convex Regime [52.459513910242485]
  We develop a general statistical framework for studying optimal noise-level allocation in diffusion training.<n>We show that the optimized training schedule admits an atomic minimizer, concentrated on finitely many noise levels.<n>We then evaluate the entropic schedule in larger-scale experiments, where full schedule optimization is currently intractable.
  arXiv  Detail & Related papers  (2026-07-10T20:59:06Z)
• STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations [46.963298896594814]
  Training Data Attribution seeks to trace a model's predictions back to its training data.<n>Most approaches approximate this effect in the parameter space using gradients.<n>We model the functional effect of training data in the activation space.
  arXiv  Detail & Related papers  (2026-06-03T17:59:36Z)
• Enhancing Eye Feature Estimation from Event Data Streams through Adaptive Inference State Space Modeling [68.1289208938377]
  Eye feature extraction from event-based data streams can be performed efficiently and with low energy consumption.<n>We introduce the emphadaptive inference state space model (AISSM), a novel architecture for feature extraction.<n>We also craft and evaluate a novel learning technique that improves training efficiency.
  arXiv  Detail & Related papers  (2026-03-14T18:47:08Z)
• Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysis [50.48301331112126]
  We propose NOVA, a training-free token reduction acceleration framework for Visual AutoRegressive modeling.<n>NOVA adaptively determines the acceleration activation scale during inference by online identifying the inflection point of scale entropy growth.<n>Experiments and analyses validate NOVA as a simple yet effective training-free acceleration framework.
  arXiv  Detail & Related papers  (2026-02-01T17:29:42Z)
• Performance Evaluation of Ising and QUBO Variable Encodings in Boltzmann Machine Learning [0.7734726150561088]
  QUBO induces larger cross terms between first- and second-order statistics, creating more small-eigenvalue directions in the Fisher information matrix.<n>Ising encoding provides more isotropic curvature and faster convergence.<n>These results clarify how representation shapes information geometry and finite-time learning dynamics in Boltzmann machines.
  arXiv  Detail & Related papers  (2025-10-15T06:57:23Z)
• Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks [3.924071936547547]
  Gated neural networks (RNNs) implicitly induce adaptive learning-rate behavior.<n>Effect arises from the coupling between state-space time scales--parametrized by the gates--and parameter-space dynamics.<n> Empirical simulations corroborate these claims.
  arXiv  Detail & Related papers  (2025-08-16T18:19:34Z)
• BEExformer: A Fast Inferencing Binarized Transformer with Early Exits [2.7651063843287718]
  We introduce Binarized Early Exit Transformer (BEExformer), the first-ever selective learning-based transformer integrating Binarization-Aware Training (BAT) with Early Exit (EE)<n>BAT employs a differentiable second-order approximation to the sign function, enabling gradient that captures both the sign and magnitude of the weights.<n>EE mechanism hinges on fractional reduction in entropy among intermediate transformer blocks with soft-routing loss estimation.<n>This accelerates inference by reducing FLOPs by 52.08% and even improves accuracy by 2.89% by resolving the "overthinking" problem inherent in deep networks.
  arXiv  Detail & Related papers  (2024-12-06T17:58:14Z)
• Rethinking PGD Attack: Is Sign Function Necessary? [131.6894310945647]
  We present a theoretical analysis of how such sign-based update algorithm influences step-wise attack performance. We propose a new raw gradient descent (RGD) algorithm that eliminates the use of sign. The effectiveness of the proposed RGD algorithm has been demonstrated extensively in experiments.
  arXiv  Detail & Related papers  (2023-12-03T02:26:58Z)
• Inference and Interference: The Role of Clipping, Pruning and Loss Landscapes in Differentially Private Stochastic Gradient Descent [13.27004430044574]
  Differentially private gradient descent (DP-SGD) is known to have poorer training and test performance on large neural networks. We compare the behavior of the two processes separately in early and late epochs. We find that while DP-SGD makes slower progress in early stages, it is the behavior in the later stages that determines the end result.
  arXiv  Detail & Related papers  (2023-11-12T13:31:35Z)
• Post-Processing Temporal Action Detection [134.26292288193298]
  Temporal Action Detection (TAD) methods typically take a pre-processing step in converting an input varying-length video into a fixed-length snippet representation sequence. This pre-processing step would temporally downsample the video, reducing the inference resolution and hampering the detection performance in the original temporal resolution. We introduce a novel model-agnostic post-processing method without model redesign and retraining.
  arXiv  Detail & Related papers  (2022-11-27T19:50:37Z)
• Learning Compact Features via In-Training Representation Alignment [19.273120635948363]
  In each epoch, the true gradient of the loss function is estimated using a mini-batch sampled from the training set. We propose In-Training Representation Alignment (ITRA) that explicitly aligns feature distributions of two different mini-batches with a matching loss. We also provide a rigorous analysis of the desirable effects of the matching loss on feature representation learning.
  arXiv  Detail & Related papers  (2022-11-23T22:23:22Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.