AdamL: A fast adaptive gradient method incorporating loss function
- URL: http://arxiv.org/abs/2312.15295v1
- Date: Sat, 23 Dec 2023 16:32:29 GMT
- Title: AdamL: A fast adaptive gradient method incorporating loss function
- Authors: Lu Xia and Stefano Massei
- Abstract summary: We propose AdamL, a novel variant of the Adam, that takes into account the loss function information to attain better results.
We show that AdamL achieves either the fastest convergence or the lowest objective function values when compared to Adam, EAdam, and AdaBelief.
In the case of vanilla convolutional neural networks, AdamL stands out from the other Adam's variants and does not require the manual adjustment of the learning rate during the later stage of the training.
- Score: 1.6025685183216696
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Adaptive first-order optimizers are fundamental tools in deep learning,
although they may suffer from poor generalization due to the nonuniform
gradient scaling. In this work, we propose AdamL, a novel variant of the Adam
optimizer, that takes into account the loss function information to attain
better generalization results. We provide sufficient conditions that together
with the Polyak-Lojasiewicz inequality, ensure the linear convergence of AdamL.
As a byproduct of our analysis, we prove similar convergence properties for the
EAdam, and AdaBelief optimizers. Experimental results on benchmark functions
show that AdamL typically achieves either the fastest convergence or the lowest
objective function values when compared to Adam, EAdam, and AdaBelief. These
superior performances are confirmed when considering deep learning tasks such
as training convolutional neural networks, training generative adversarial
networks using vanilla convolutional neural networks, and long short-term
memory networks. Finally, in the case of vanilla convolutional neural networks,
AdamL stands out from the other Adam's variants and does not require the manual
adjustment of the learning rate during the later stage of the training.
Related papers
- AdamZ: An Enhanced Optimisation Method for Neural Network Training [1.54994260281059]
AdamZ dynamically adjusts the learning rate by incorporating mechanisms to address overshooting and stagnation.
It consistently excels in minimising the loss function, making it particularly advantageous for applications where precision is critical.
arXiv Detail & Related papers (2024-11-22T23:33:41Z) - Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning [2.695991050833627]
We propose a new optimization algorithm named CG-like-Adam for deep learning.
Specifically, both the first-order and the second-order moment estimation of generic Adam are replaced by the conjugate-gradient-like.
Numerical experiments show the superiority of the proposed algorithm based on the CIFAR10/100 dataset.
arXiv Detail & Related papers (2024-04-02T07:57:17Z) - Robust Learning with Progressive Data Expansion Against Spurious
Correlation [65.83104529677234]
We study the learning process of a two-layer nonlinear convolutional neural network in the presence of spurious features.
Our analysis suggests that imbalanced data groups and easily learnable spurious features can lead to the dominance of spurious features during the learning process.
We propose a new training algorithm called PDE that efficiently enhances the model's robustness for a better worst-group performance.
arXiv Detail & Related papers (2023-06-08T05:44:06Z) - Theoretical Characterization of the Generalization Performance of
Overfitted Meta-Learning [70.52689048213398]
This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features.
We find new and interesting properties that do not exist in single-task linear regression.
Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
arXiv Detail & Related papers (2023-04-09T20:36:13Z) - Lipschitzness Effect of a Loss Function on Generalization Performance of
Deep Neural Networks Trained by Adam and AdamW Optimizers [0.0]
We theoretically prove that the Lipschitz constant of a loss function is an important factor to diminish the generalization error of the output model obtained by Adam or AdamW.
Our experimental evaluation shows that the loss function with a lower Lipschitz constant and maximum value improves the generalization of the model trained by Adam or AdamW.
arXiv Detail & Related papers (2023-03-29T05:33:53Z) - Theoretical Characterization of How Neural Network Pruning Affects its
Generalization [131.1347309639727]
This work makes the first attempt to study how different pruning fractions affect the model's gradient descent dynamics and generalization.
It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero.
More surprisingly, the generalization bound gets better as the pruning fraction gets larger.
arXiv Detail & Related papers (2023-01-01T03:10:45Z) - Understanding the Generalization of Adam in Learning Neural Networks
with Proper Regularization [118.50301177912381]
We show that Adam can converge to different solutions of the objective with provably different errors, even with weight decay globalization.
We show that if convex, and the weight decay regularization is employed, any optimization algorithms including Adam will converge to the same solution.
arXiv Detail & Related papers (2021-08-25T17:58:21Z) - Towards an Understanding of Benign Overfitting in Neural Networks [104.2956323934544]
Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss.
We examine how these benign overfitting phenomena occur in a two-layer neural network setting.
We show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate.
arXiv Detail & Related papers (2021-06-06T19:08:53Z) - Towards Practical Adam: Non-Convexity, Convergence Theory, and
Mini-Batch Acceleration [12.744658958445024]
Adam is one of the most influential adaptive algorithms for training deep neural networks.
Existing approaches, such as decreasing an adaptive learning rate, adopting a big batch size, have tried to promote Adam-type algorithms to converge.
We introduce an alternative easy-to-check sufficient condition, which merely depends on the parameters of historical base learning rate.
arXiv Detail & Related papers (2021-01-14T06:42:29Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.