Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data

• URL: http://arxiv.org/abs/2607.06013v1
• Date: Tue, 07 Jul 2026 08:55:35 GMT
• Title: Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data
• Abstract summary: This paper studies the rate-controlled middle case for full-batch linear classification on separable data.<n>For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice.
• Score: 3.7437735326653883
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adaptive gradient methods can favor max-margin separators that differ from gradient descent, yet a fixed positive numerical stability constant eventually changes the update geometry again. This paper studies the rate-controlled middle case for full-batch linear classification on separable data. For memoryless stability-annealed smoothed-sign descent with weighted exponential loss, we prove that the normalized iterates converge to the minimizer of a convex Burg-type barrier over a margin slice. The proof rewrites the dynamics exactly as entropic mirror ascent on a concave dual objective, controls the dual gap by a KL recursion, and yields an explicit S_t^{-1/2} normalized-iterate envelope. The static barrier geometry is fully characterized, including KKT conditions and both endpoint limits. Experiments validate the exact dual identities to floating-point error, illustrate the predicted path and rate diagram, and show an empirical fixed-epsilon crossover scaling in cumulative time. We further report robustness and boundary diagnostics for logistic tails, fixed-epsilon crossover, and adaptive-method variants, delineating the scope of the proved smoothed-sign theory.

Related papers

• CyclOT: Learning Quadratic Optimal Transport Maps via Synchronized Forward-Backward Interpolants [20.064386074670605]
  We introduce a bidirectional neural framework in which the learned maps induce forward and backward displacement interpolants.<n>Experiments on Swiss roll, MNIST, CelebA, single-cell data, and chest X-ray images evaluate endpoint fidelity, transport cost inverse, consistency, and the geometry of the induced inverts.
  arXiv  Detail & Related papers  (2026-09-12T11:31:53Z)
• Explaining f-Divergence-Based Regularization via Local Curvature and Sharpness-Aware Minimization [7.792020418343022]
  Divergence-based regularization and Sharpness-Aware Minimization (SAM) are two prominent approaches for improving generalization in deep learning.<n>We show that the two methods are locally consistent under parameter-space perturbations.<n>We also show that divergence-based regularization is defined through transformations of the input.
  arXiv  Detail & Related papers  (2026-09-08T19:01:05Z)
• Disentangling Continuous-Time Latent Dynamics: Identifiability of Latent SDEs via Diffusion Shifts [60.95873409673775]
  identifiability in continuous-time latent differential equation models remains largely open.<n>We study additive-noise latent SDEs observed through an unknown nonlinear diffeomorphism.<n>We propose a two-stage estimator for latent disentanglement and optional graph recovery.
  arXiv  Detail & Related papers  (2026-06-26T16:18:28Z)
• Why Zeroth-Order Adaptation May Forget Less: A Randomized Shaping Theory [23.912005398880293]
  Continual learning requires new-task adaptation without damaging previously acquired capabilities.<n>Recent forward-pass and zeroth-order (ZO) results show that low-query adaptation may retain better than first-order descent.<n>We give a local randomized gradient-shaping analysis: finite differences expose a raw shape that is mean-aligned with FO.<n>For norm-matched ZO, the expected shaped retention curvature obeys an exact identity that preserves the isotropic retention floor while contracting only the anisotropic component.<n>Projecting this identity onto the incoming gradient yields the observable FO--ZO quadratic forgetting gap.
  arXiv  Detail & Related papers  (2026-05-11T14:41:08Z)
• Direct Estimation of Schrödinger Bridge Time-Series Drifts: Finite-Sample, Asymptotic, and Adaptive Guarantees [0.0]
  We study nonparametric estimation of Schrdinger bridge drifts from i.i.d. data.<n>Our approach works directly at the drift level and isolates emphstatistical error from optimization, approximation, and discretization error.
  arXiv  Detail & Related papers  (2026-05-06T20:44:44Z)
• From Video-to-PDE: Data-Driven Discovery of Nonlinear Dye Plume Dynamics [0.0]
  Inferring continuum models directly from video is hampered by two facts: the recorded field is uncalibrated image intensity rather than a physical state.<n>We develop a video-to-PDE pipeline that converts grayscale recordings of an ink plume into a normalised scalar field.<n>We show that uncalibrated visual data can yield compact, predictive and structurally interpretable continuum models when discovery, calibration and uncertainty are treated as distinct stages.
  arXiv  Detail & Related papers  (2026-05-06T06:23:59Z)
• Efficient Inference after Directionally Stable Adaptive Experiments [47.32051320630248]
  We study inference on pathwise differentiable targets after adaptive data collection, such as a bandit.<n>We introduce a novel target-specific condition, directional stability, which is strictly weaker than previously imposed target-aparametric stability conditions.
  arXiv  Detail & Related papers  (2026-02-25T01:09:18Z)
• Distributional Stability of Tangent-Linearized Gaussian Inference on Smooth Manifolds [3.632189127068905]
  We derive explicit non-asymptotic $W$ stability bounds for projection marginalization and surface-measure conditioning.<n>The bounds separate local second-order geometric distortion from nonlocal tail leakage.<n>Experiments validate the predicted calibration transition near $sqrt||_mathrmop/Rapprox 1/6$ and indicate that normal-direction uncertainty is the dominant failure mode when locality breaks.
  arXiv  Detail & Related papers  (2026-02-22T13:18:45Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability [69.01076284478151]
  In machine learning optimization, gradient descent (GD) often operates at the edge of stability (EoS) This paper studies the convergence and implicit bias of constant-stepsize GD for logistic regression on linearly separable data in the EoS regime.
  arXiv  Detail & Related papers  (2023-05-19T16:24:47Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• The Convex Geometry of Backpropagation: Neural Network Gradient Flows Converge to Extreme Points of the Dual Convex Program [26.143558180103334]
  We study non- subgradient flows for two-layer ReLULU networks from a convex implicit geometry and duality perspective. We show that we can identify the problem of non- subgradient descent via primal-dual correspondence.
  arXiv  Detail & Related papers  (2021-10-13T04:17:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.