RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning

• URL: http://arxiv.org/abs/2607.19544v1
• Date: Tue, 21 Jul 2026 19:43:54 GMT
• Title: RELTA-SGLD: Relative-Growth Localized Taming for Nonconvex Stochastic-Gradient Langevin Learning
• Abstract summary: We introduce RELTA-SGLD, a taming scheme that stabilizes superlinear-gradient updates while reducing unnecessary suppression of the original learning.<n>A threshold determines where the taming turns on, while a relative-growth principle derived from the one-step Lyapunov condition determines the taming strength.
• Score: 10.599439539657785
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce RELTA-SGLD, a taming scheme that stabilizes superlinear stochastic-gradient updates while reducing unnecessary suppression of the original learning drift. A threshold determines where the taming turns on, while a relative-growth principle derived from the one-step Lyapunov stability condition determines the required taming strength. Together, they produce a lighter $λ$-scale denominator and preserve a nonvanishing far-tail return. As a consequence, we prove polynomial moment stability and first-order stationary accuracy in both $W_1$ and $W_2$ for nonconvex SGLD with superlinearly growing stochastic-gradient oracles, improving the corresponding half-order and quarter-order bounds for comparable stochastic-gradient tamed schemes. On Fashion-MNIST under active stabilization pressure, RELTA improves the mean learning metrics over both untamed SGLD and TUSLA and remains competitive with a tuned AdamW reference. In an ordinary-training regime, its lighter localized denominator reduces unnecessary perturbation of the original update and maintains nearly untamed learning dynamics.

Related papers

• The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization [50.59470683896735]
  min-max optimization has attracted increasing attention due to its applications in modern machine learning.<n>We provide a comprehensive theoretical of gradient descent (SGDA)<n>We develop new clipping-free algorithms, i.e., Stoc-GDAM and Stoc-TRmax, and they both can achieve the optimal dependence on the target accuracy without using gradient clipping.
  arXiv  Detail & Related papers  (2026-09-05T12:49:28Z)
• The Road Taken: The Role of Optimizers at the Edge of Stability [53.81268610971847]
  edge of stability refers to a phenomenon in deep learning where the Hessian eigenvalues of the loss remain stable above a threshold that the classical descent predicts to be unstable.<n>Previous works formulate the edge of stability with respect to the maximum Hessian eigenvalue and the learning rate.<n>We observe that many first-order methods, including gradient descent, significantly violate the stability bound predicted by these theories by a factor as large as $times 21.1$.
  arXiv  Detail & Related papers  (2026-08-19T01:03:34Z)
• TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs [83.78708832901194]
  Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild.<n>However, real-world deployment remains challenging: modalities may be missing or noisy at test time.<n>We propose a solution in which a frozen teacher guides an adaptive low-rank student via self-distillation.<n>Experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation.
  arXiv  Detail & Related papers  (2026-08-18T23:40:28Z)
• Staleness-Learning Rate Scaling Laws for Asynchronous RLHF [18.98349589222509]
  We show that stale rollouts introduce a per-step surrogate-gradient bias of order O(S * eta), where S denotes the maximum rollout lag and eta denotes the learning rate.<n>This explains why the maximum stable learning rate may appear weakly dependent on staleness in the horizon-limited regime.
  arXiv  Detail & Related papers  (2026-07-01T15:40:12Z)
• SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates [2.104588729923885]
  Modern deep learning has been shown to operate at the edge of stability.<n>We provide convergence guarantees for Gradient Descent (SGD) applied to the multiclass cross-entropy loss.<n> Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime.
  arXiv  Detail & Related papers  (2026-06-29T21:32:58Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Momentum Further Constrains Sharpness at the Edge of Stochastic Stability [0.5405981353784005]
  Recent work suggests (stochastic) gradient descent self-organizes near an instability boundary, shaping both optimization and the solutions found.<n>We demonstrate that SGD with momentum exhibits an Edge of Stability (EoSS)-like regime with batch-size-dependent behavior.
  arXiv  Detail & Related papers  (2026-04-15T17:28:46Z)
• A Unified Noise-Curvature View of Loss of Trainability [8.602734307457387]
  Loss of trainability (LoT) in continual learning occurs when steps no longer yield improvement as tasks evolve.<n>We introduce two complementary criteria: a batch-size-aware gradient-noise bound and a curvature volatility-controlled bound.<n>Using this threshold, we build a simple per-layer scheduler that keeps each layers effective step below a safe limit.
  arXiv  Detail & Related papers  (2025-09-24T02:11:13Z)
• Stability and Generalization for Bellman Residuals [8.250374560598493]
  Bellman residual minimization (BRM) has emerged as an attractive remedy for offline reinforcement learning.<n>This paper introduces a single Lyapunov potential that couples SGDA runs on neighbouring datasets and yields an O (1/n) on-average argument-stability bound.<n>The results hold for standard neural-network parameterizations and minibatch SGD.
  arXiv  Detail & Related papers  (2025-08-26T07:15:36Z)
• Convergence of mean-field Langevin dynamics: Time and space discretization, stochastic gradient, and variance reduction [49.66486092259376]
  The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin dynamics that incorporates a distribution-dependent drift. Recent works have shown that MFLD globally minimizes an entropy-regularized convex functional in the space of measures. We provide a framework to prove a uniform-in-time propagation of chaos for MFLD that takes into account the errors due to finite-particle approximation, time-discretization, and gradient approximation.
  arXiv  Detail & Related papers  (2023-06-12T16:28:11Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.