Accelerated Gradient Descent for Faster Convergence with Minimal Overhead

• URL: http://arxiv.org/abs/2605.16017v1
• Date: Fri, 15 May 2026 14:50:39 GMT
• Title: Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
• Abstract summary: We present CTAGD (Curature-Tuned Accelerated Gradient), an optimization method for non-problems in deep learning tasks.<n>CTAGD is a general boosting procedure that accelerates first-order optimization methods by explicitly using finite-difference quotients.<n>Experiments demonstrate that CT-AGD achieves same accuracy level as baseline first-order methods, yet reduces the required epochs by 33% on average.
• Score: 1.9686770963118387
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we present CT-AGD (Curvature-Tuned Accelerated Gradient Descent), an optimization method for non-convex optimization problems in deep learning training tasks. CT-AGD is a general boosting procedure that accelerates first-order methods by explicitly capturing the local curvature using finite-difference quotients, and the development of heuristics aimed at mitigating noise and bias introduced by stochastic mini-batch training. CT-AGD has a comparable storage and computational overhead as adaptive gradient methods such as Adam. Our extensive experiments demonstrate that CT-AGD achieves the same level of accuracy as the baseline first-order methods, yet reduces the required training epochs by 33% on average.

Related papers

• Multilevel Stochastic Plug-and-Play for Sparse-View CT Reconstruction [42.351825574378395]
  Sparse-view computed tomography (SVCT) reduces radiation and time, but the limited number of projection views makes the reconstruction problem severely ill-posed and leads to streak artifacts.<n>Experiments on SVCT reconstruction show that our method, called Multilevel PlugandPlay (ML-S), achieves reconstruction quality comparable to state-and-the-art methods.
  arXiv  Detail & Related papers  (2026-06-23T13:34:30Z)
• Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization [1.6749379740049926]
  We introduce F-CMA, a Fast-Controlled Mini-batch Algorithm with a random reshuffling method featuring a sufficient decrease condition and a line-search procedure to ensure loss reduction per epoch.<n>Tests show significant improvements, including a decrease in the overall training time by 68%, an increase in per-epoch efficiency by up to 20%, and in model accuracy by up to 5%.
  arXiv  Detail & Related papers  (2024-11-24T11:46:47Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Inverse-Free Fast Natural Gradient Descent Method for Deep Learning [52.0693420699086]
  We present a fast natural gradient descent (FNGD) method that only requires inversion during the first epoch. FNGD exhibits similarities to the average sum in first-order methods, leading to the computational complexity of FNGD being comparable to that of first-order methods.
  arXiv  Detail & Related papers  (2024-03-06T05:13:28Z)
• Faster One-Sample Stochastic Conditional Gradient Method for Composite Convex Minimization [61.26619639722804]
  We propose a conditional gradient method (CGM) for minimizing convex finite-sum objectives formed as a sum of smooth and non-smooth terms. The proposed method, equipped with an average gradient (SAG) estimator, requires only one sample per iteration. Nevertheless, it guarantees fast convergence rates on par with more sophisticated variance reduction techniques.
  arXiv  Detail & Related papers  (2022-02-26T19:10:48Z)
• A New Adaptive Gradient Method with Gradient Decomposition [3.04585143845864]
  We propose a new adaptive method called DecGD which generalizes poorly compared with gradient descent (SGD) and its accelerated schemes (SGDM) Our method adjusts the learning rates adaptively according to the current loss based vector instead of the squared gradient used in Adam-type methods. We show that DecGD shows better generalization performance than SGDM and rapid convergence like Adam-type methods.
  arXiv  Detail & Related papers  (2021-07-18T06:37:28Z)
• Adapting Stepsizes by Momentumized Gradients Improves Optimization and Generalization [89.66571637204012]
  textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing.
  arXiv  Detail & Related papers  (2021-06-22T03:13:23Z)
• Adaptive Gradient Methods Can Be Provably Faster than SGD after Finite Epochs [25.158203665218164]
  We show that adaptive gradient methods can be faster than random shuffling SGD after finite time. To the best of our knowledge, it is the first to demonstrate that adaptive gradient methods can be faster than SGD after finite time.
  arXiv  Detail & Related papers  (2020-06-12T09:39:47Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets [71.05306664267832]
  Adaptive algorithms perform gradient updates using the history of gradients and are ubiquitous in training deep neural networks. In this paper we analyze a variant of OptimisticOA algorithm for nonconcave minmax problems. Our experiments show that adaptive GAN non-adaptive gradient algorithms can be observed empirically.
  arXiv  Detail & Related papers  (2019-12-26T22:10:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.