Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity

• URL: http://arxiv.org/abs/2607.22906v1
• Date: Fri, 24 Jul 2026 20:50:13 GMT
• Title: Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity
• Abstract summary: We find that one-sided Hlder is an effective adaptive step-size signal when full-gradient variation is inflated by directions that do not inflate it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions that do not decrease it by directions
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided Hölder regularity. Unlike classical Hölder- or Lipschitz-gradient assumptions, which control the full gradient variation, our condition bounds only the directional term appearing in the descent inequality. This can allow less conservative step sizes when large gradient changes are orthogonal to, or favorable along, the update direction. We propose an adaptive scalar-step method based on an estimate of positive one-sided Hölder curvature, combined with a simple sufficient-decrease safeguard. For nonconvex objectives on a convex region containing the accepted update segments, we prove an explicit best-iterate stationarity bound with a rate determined by the Hölder exponent. Unlike predetermined diminishing step-size schemes, the method adapts to the local descent geometry. We evaluate the approach on two full-batch benchmarks designed to separate directional curvature from full gradient variation. On a binary classification problem, the method achieves the lowest final cross-entropy, objective value, and gradient norm, together with the largest classification margin among the compared scalar gradient methods. On a nonconvex Hölder regression problem, it attains the lowest final objective gap and gradient norm. These results indicate that one-sided Hölder curvature is an effective adaptive step-size signal when full-gradient variation is inflated by directions that do not hinder descent.

Related papers

• Semiparametric Efficient Bilevel Gradient Estimation [40.349437034326606]
  We develop a semiparametric debiasing theory for population bilevels based on the efficient influence function.<n>We establish a cross-fitted hypergradient estimator for which we establish normality together with uniform control over the outer parameter.<n>On synthetic bilevel benchmarks with known ground truth, the method tracks the efficient-gradient benchmark and improves over plug-in functional hypergradients and regularized kernel bilevel baselines.
  arXiv  Detail & Related papers  (2026-05-20T16:07:55Z)
• Flatness and Gradient Alignment Are Both Necessary: Spectral-Aware Gradient-Aligned Exploration for Multi-Distribution Learning [7.794885131732119]
  We show that flatness and gradient alignment should be considered in multi-distribution learning settings.<n>We propose SAGE (Spectral-Aware Gradient-Aligned Exploration) that targets both terms.<n> Experiments on five domain-generalization and two multi-task learning benchmarks show that the proposed method establishes a new state-of-the-art.
  arXiv  Detail & Related papers  (2026-05-08T15:52:34Z)
• First-ish Order Methods: Hessian-aware Scalings of Gradient Descent [11.125968799758436]
  A key limitation of gradient descent is its lack of natural scaling.<n>By accounting for curvature, our adaptive, Hessian-aware scaling method ensures a local unit step size.<n>We show that our method converges globally under a significantly weaker version of the standard Lipschitz assumption.
  arXiv  Detail & Related papers  (2025-02-06T01:22:23Z)
• Gradient Equilibrium in Online Learning: Theory and Applications [56.02856551198923]
  gradient equilibrium is achieved by standard online learning methods.<n> gradient equilibrium translates into an interpretable and meaningful property in online prediction problems.<n>We show that gradient equilibrium framework can be used to develop a debiasing scheme for black-box predictions.
  arXiv  Detail & Related papers  (2025-01-14T18:59:09Z)
• Directional Smoothness and Gradient Methods: Convergence and Adaptivity [16.779513676120096]
  We develop new sub-optimality bounds for gradient descent that depend on the conditioning of the objective along the path of optimization.<n>Key to our proofs is directional smoothness, a measure of gradient variation that we use to develop upper-bounds on the objective.<n>We prove that the Polyak step-size and normalized GD obtain fast, path-dependent rates despite using no knowledge of the directional smoothness.
  arXiv  Detail & Related papers  (2024-03-06T22:24:05Z)
• Why is parameter averaging beneficial in SGD? An objective smoothing perspective [13.863368438870562]
  gradient descent (SGD) and its implicit bias are often characterized in terms of the sharpness of the minima. We study the commonly-used averaged SGD algorithm, which has been empirically observed in Izmailov et al. We prove that averaged SGD can efficiently optimize the smoothed objective which avoids sharp local minima.
  arXiv  Detail & Related papers  (2023-02-18T16:29:06Z)
• The Equalization Losses: Gradient-Driven Training for Long-tailed Object Recognition [84.51875325962061]
  We propose a gradient-driven training mechanism to tackle the long-tail problem. We introduce a new family of gradient-driven loss functions, namely equalization losses. Our method consistently outperforms the baseline models.
  arXiv  Detail & Related papers  (2022-10-11T16:00:36Z)
• Decreasing scaling transition from adaptive gradient descent to stochastic gradient descent [1.7874193862154875]
  We propose a decreasing scaling transition from adaptive gradient descent to gradient descent method DSTAda. Our experimental results show that DSTAda has a faster speed, higher accuracy, and better stability and robustness.
  arXiv  Detail & Related papers  (2021-06-12T11:28:58Z)
• A Study of Gradient Variance in Deep Learning [56.437755740715396]
  We introduce a method, Gradient Clustering, to minimize the variance of average mini-batch gradient with stratified sampling. We measure the gradient variance on common deep learning benchmarks and observe that, contrary to common assumptions, gradient variance increases during training.
  arXiv  Detail & Related papers  (2020-07-09T03:23:10Z)
• Carath\'eodory Sampling for Stochastic Gradient Descent [79.55586575988292]
  We present an approach that is inspired by classical results of Tchakaloff and Carath'eodory about measure reduction. We adaptively select the descent steps where the measure reduction is carried out. We combine this with Block Coordinate Descent so that measure reduction can be done very cheaply.
  arXiv  Detail & Related papers  (2020-06-02T17:52:59Z)
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets [71.05306664267832]
  Adaptive algorithms perform gradient updates using the history of gradients and are ubiquitous in training deep neural networks. In this paper we analyze a variant of OptimisticOA algorithm for nonconcave minmax problems. Our experiments show that adaptive GAN non-adaptive gradient algorithms can be observed empirically.
  arXiv  Detail & Related papers  (2019-12-26T22:10:10Z)
• On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization [80.03647903934723]
  We prove adaptive gradient methods in expectation of gradient convergence methods. Our analyses shed light on better adaptive gradient methods in optimizing non understanding gradient bounds.
  arXiv  Detail & Related papers  (2018-08-16T20:25:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.