Beta-Scheduling: Momentum from Critical Damping as a Diagnostic and Correction Tool for Neural Network Training

• URL: http://arxiv.org/abs/2603.28921v2
• Date: Sun, 05 Apr 2026 21:20:15 GMT
• Title: Beta-Scheduling: Momentum from Critical Damping as a Diagnostic and Correction Tool for Neural Network Training
• Abstract summary: Standard neural network training uses constant momentum (typically 0.9), a convention dating to 1964 with limited theoretical justification for its optimality.<n>This beta-schedule requires zero free parameters beyond the existing learning rate schedule.<n>On ResNet-18/CIFAR-10, beta-scheduling delivers 1.9x faster convergence to 90% accuracy compared to constant momentum.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard neural network training uses constant momentum (typically 0.9), a convention dating to 1964 with limited theoretical justification for its optimality. We derive a time-varying momentum schedule from the critically damped harmonic oscillator: mu(t) = 1 - 2*sqrt(alpha(t)), where alpha(t) is the current learning rate. This beta-schedule requires zero free parameters beyond the existing learning rate schedule. On ResNet-18/CIFAR-10, beta-scheduling delivers 1.9x faster convergence to 90% accuracy compared to constant momentum. More importantly, the per-layer gradient attribution under this schedule produces a cross-optimizer invariant diagnostic: the same three problem layers are identified regardless of whether the model was trained with SGD or Adam (100% overlap). Surgical correction of only these layers fixes 62 misclassifications while retraining only 18% of parameters. A hybrid schedule -- physics momentum for fast early convergence, then constant momentum for the final refinement -- reaches 95% accuracy fastest among five methods tested. The main contribution is not an accuracy improvement but a principled, parameter-free tool for localizing and correcting specific failure modes in trained networks.

Related papers

• SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning [70.71795707610961]
  method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
  arXiv  Detail & Related papers  (2026-08-29T21:34:47Z)
• Propensity Straight-Through Gradients for Discrete Stochastic Systems [0.0]
  We exploit the affine state update to obtain the exact one-step conditional-mean sensitivity.<n>We show that one-step sensitivities composed across events can depart from the exact multistep sensitivity.
  arXiv  Detail & Related papers  (2026-08-26T10:59:25Z)
• Predicting Deep Neural Network Training Outcomes from Early Training Telemetry [2.310157248304176]
  We study whether a single training run's own early telemetry can predict that run's eventual outcome without reference to other runs.<n>Across 23,788 training runs spanning six architecture/dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R2 = 0.92-0.99 for final-accuracy regression.<n>These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation.
  arXiv  Detail & Related papers  (2026-08-04T14:13:21Z)
• Self-Adversarial One Step Generation via Condition Shifting [11.426065945265647]
  APEX is a framework for adversarial correction signals extracted endogenously from a flow model.<n>Our 0.6B model surpasses FLUX-Schnell 12B (20$times$ more parameters) in one step quality.<n>With LoRA tuning on Qwen-Image 20B, APEX reaches a GenEval score of 0.89 at NFE=1 in 6 hours, surpassing the original 50-step teacher (0.87) and providing a 15.33$times$ inference speedup.
  arXiv  Detail & Related papers  (2026-04-14T05:54:33Z)
• Robust Physics Discovery from Highly Corrupted Data: A PINN Framework Applied to the Nonlinear Schrödinger Equation [0.0]
  We demonstrate a deep learning framework capable of recovering physical parameters from the NVIDIA Schrodinger Equation (LSEN) under severe noise conditions.<n>We achieve reconstruction of the nonlinear coefficient beta with less than 0.2 percent relative error using only 500 sparse, randomly sampled data points.<n>Our results indicate that physics-based regularization acts as an effective filter against high measurement uncertainty.
  arXiv  Detail & Related papers  (2026-01-07T18:43:11Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• Beyond the Ideal: Analyzing the Inexact Muon Update [54.70108543057578]
  We show first analysis of the inexactized update at Muon's core.<n>We reveal a fundamental coupling between this inexactness and the optimal step size and momentum.
  arXiv  Detail & Related papers  (2025-10-22T18:01:07Z)
• Assessing the Limits of Graph Neural Networks for Vapor-Liquid Equilibrium Prediction: A Cryogenic Mixture Case Study [3.765010805872486]
  This study asks whether a structure-aware graph neural network (GNN; DimeNet++) trained on GERG-2008/CoolProp data can act as a practical surrogate for an equation of state (EoS)<n>We generate a ternary dataset over 90-200 K and pressures to 100 bar, curate it with a 15% density filter (reducing 5,200 states to 1,516), and pair each state with a lightweight molecular-dynamics snapshot to supply structural features.<n>We conclude that, as configured, the surrogate in this study is not equilibrium-ready for VLE and offers no runtime benefit.
  arXiv  Detail & Related papers  (2025-09-10T16:10:58Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We present a novel variational formulation of the calibration-refinement decomposition.<n>We provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Efficiently Training Time-to-First-Spike Spiking Neural Networks from Scratch [39.05124192217359]
  Spiking Neural Networks (SNNs) are well-suited for energy-efficient neuromorphic hardware.<n>Time-to-First-Spike (TTFS) coding, which uses a single spike per neuron, offers extreme sparsity and energy efficiency but suffers from unstable training and low accuracy due to its sparse firing.<n>We propose a training framework incorporating parameter normalization, training normalization, temporal output decoding, and pooling layer re-evaluation.<n>Experiments show the framework stabilizes and accelerates training, reduces latency, and achieves state-of-the-art accuracy for TTFS SNNs on M
  arXiv  Detail & Related papers  (2024-10-31T04:14:47Z)
• DRIVE: Dual Gradient-Based Rapid Iterative Pruning [2.209921757303168]
  Modern deep neural networks (DNNs) consist of millions of parameters, necessitating high-performance computing during training and inference. Traditional pruning methods that are applied post-training focus on streamlining inference, but there are recent efforts to leverage sparsity early on by pruning before training. We present Dual Gradient-Based Rapid Iterative Pruning (DRIVE), which leverages dense training for initial epochs to counteract the randomness inherent at the inception.
  arXiv  Detail & Related papers  (2024-04-01T20:44:28Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)
• Flatter, faster: scaling momentum for optimal speedup of SGD [0.0]
  We study training dynamics arising from interplay between gradient descent (SGD) and label noise and momentum in the training of neural networks. We find that scaling the momentum hyper parameter $1-NISTbeta$ with the learning rate to the power of $2/3$ maximally accelerates training, without sacrificing generalization.
  arXiv  Detail & Related papers  (2022-10-28T20:41:48Z)
• Distribution Mismatch Correction for Improved Robustness in Deep Neural Networks [86.42889611784855]
  normalization methods increase the vulnerability with respect to noise and input corruptions. We propose an unsupervised non-parametric distribution correction method that adapts the activation distribution of each layer. In our experiments, we empirically show that the proposed method effectively reduces the impact of intense image corruptions.
  arXiv  Detail & Related papers  (2021-10-05T11:36:25Z)
• Fast Certified Robust Training via Better Initialization and Shorter Warmup [95.81628508228623]
  We propose a new IBP and principled regularizers during the warmup stage to stabilize certified bounds. We find that batch normalization (BN) is a crucial architectural element to build best-performing networks for certified training.
  arXiv  Detail & Related papers  (2021-03-31T17:58:58Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.