Improving Gradient-Trend Identification: Fast-Adaptive Moment Estimation
with Finance-Inspired Triple Exponential Moving Average
- URL: http://arxiv.org/abs/2306.01423v2
- Date: Thu, 21 Dec 2023 08:39:17 GMT
- Title: Improving Gradient-Trend Identification: Fast-Adaptive Moment Estimation
with Finance-Inspired Triple Exponential Moving Average
- Authors: Roi Peleg, Teddy Lazebnik, Assaf Hoogi
- Abstract summary: We introduce a novel called fast-adaptive moment estimation (FAME)
Inspired by the triple exponential moving average (TEMA) used in the financial domain, FAME improves the precision of identifying gradient trends.
Because of the introduction of TEMA into the optimization process, FAME can identify trends with higher accuracy and fewer lag issues.
- Score: 2.480023305418
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The performance improvement of deep networks significantly depends on their
optimizers. With existing optimizers, precise and efficient recognition of the
gradients trend remains a challenge. Existing optimizers predominantly adopt
techniques based on the first-order exponential moving average (EMA), which
results in noticeable delays that impede the real-time tracking of gradients
trend and consequently yield sub-optimal performance. To overcome this
limitation, we introduce a novel optimizer called fast-adaptive moment
estimation (FAME). Inspired by the triple exponential moving average (TEMA)
used in the financial domain, FAME leverages the potency of higher-order TEMA
to improve the precision of identifying gradient trends. TEMA plays a central
role in the learning process as it actively influences optimization dynamics;
this role differs from its conventional passive role as a technical indicator
in financial contexts. Because of the introduction of TEMA into the
optimization process, FAME can identify gradient trends with higher accuracy
and fewer lag issues, thereby offering smoother and more consistent responses
to gradient fluctuations compared to conventional first-order EMA. To study the
effectiveness of our novel FAME optimizer, we conducted comprehensive
experiments encompassing six diverse computer-vision benchmarks and tasks,
spanning detection, classification, and semantic comprehension. We integrated
FAME into 15 learning architectures and compared its performance with those of
six popular optimizers. Results clearly showed that FAME is more robust and
accurate and provides superior performance stability by minimizing noise (i.e.,
trend fluctuations). Notably, FAME achieves higher accuracy levels in
remarkably fewer training epochs than its counterparts, clearly indicating its
significance for optimizing deep networks in computer-vision tasks.
Related papers
- Variational Stochastic Gradient Descent for Deep Neural Networks [16.96187187108041]
Current state-of-the-arts are adaptive gradient-based optimization methods such as Adam.
Here, we propose to combine both approaches, resulting in the Variational Gradient Descent (VSGD)
We show how our VSGD method relates to other adaptive gradient-baseds like Adam.
arXiv Detail & Related papers (2024-04-09T18:02:01Z) - Online Adaptive Disparity Estimation for Dynamic Scenes in Structured
Light Systems [17.53719804060679]
Self-supervised online adaptation has been proposed as a solution to bridge this performance gap.
We propose an unsupervised loss function based on long sequential inputs. It ensures better gradient directions and faster convergence.
Our proposed framework significantly improves the online adaptation speed and achieves superior performance on unseen data.
arXiv Detail & Related papers (2023-10-13T08:00:33Z) - We Don't Need No Adam, All We Need Is EVE: On The Variance of Dual
Learning Rate And Beyond [0.0]
This paper introduces a novel method, Enhanced Velocity Estimation (EVE), which innovatively applies different learning rates to distinct components of the gradients.
By bifurcating the learning rate, EVE enables more nuanced control and faster convergence, addressing the challenges associated with traditional single learning rate approaches.
arXiv Detail & Related papers (2023-08-21T14:08:42Z) - Multiplicative update rules for accelerating deep learning training and
increasing robustness [69.90473612073767]
We propose an optimization framework that fits to a wide range of machine learning algorithms and enables one to apply alternative update rules.
We claim that the proposed framework accelerates training, while leading to more robust models in contrast to traditionally used additive update rule.
arXiv Detail & Related papers (2023-07-14T06:44:43Z) - Learning Large-scale Neural Fields via Context Pruned Meta-Learning [60.93679437452872]
We introduce an efficient optimization-based meta-learning technique for large-scale neural field training.
We show how gradient re-scaling at meta-test time allows the learning of extremely high-quality neural fields.
Our framework is model-agnostic, intuitive, straightforward to implement, and shows significant reconstruction improvements for a wide range of signals.
arXiv Detail & Related papers (2023-02-01T17:32:16Z) - Improving Multi-fidelity Optimization with a Recurring Learning Rate for
Hyperparameter Tuning [7.591442522626255]
We propose Multi-fidelity Optimization with a Recurring Learning rate (MORL)
MORL incorporates CNNs' optimization process into multi-fidelity optimization.
It alleviates the problem of slow-starter and achieves a more precise low-fidelity approximation.
arXiv Detail & Related papers (2022-09-26T08:16:31Z) - Accelerating Federated Learning with a Global Biased Optimiser [16.69005478209394]
Federated Learning (FL) is a recent development in the field of machine learning that collaboratively trains models without the training data leaving client devices.
We propose a novel, generalised approach for applying adaptive optimisation techniques to FL with the Federated Global Biased Optimiser (FedGBO) algorithm.
FedGBO accelerates FL by applying a set of global biased optimiser values during the local training phase of FL, which helps to reduce client-drift' from non-IID data.
arXiv Detail & Related papers (2021-08-20T12:08:44Z) - Adaptive Gradient Method with Resilience and Momentum [120.83046824742455]
We propose an Adaptive Gradient Method with Resilience and Momentum (AdaRem)
AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient.
Our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error.
arXiv Detail & Related papers (2020-10-21T14:49:00Z) - Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
We show that a host of variations can be covered in a unified framework that we propose.
We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
arXiv Detail & Related papers (2020-06-10T08:22:41Z) - The Break-Even Point on Optimization Trajectories of Deep Neural
Networks [64.7563588124004]
We argue for the existence of the "break-even" point on this trajectory.
We show that using a large learning rate in the initial phase of training reduces the variance of the gradient.
We also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers.
arXiv Detail & Related papers (2020-02-21T22:55:51Z) - Large Batch Training Does Not Need Warmup [111.07680619360528]
Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications.
In this paper, we propose a novel Complete Layer-wise Adaptive Rate Scaling (CLARS) algorithm for large-batch training.
Based on our analysis, we bridge the gap and illustrate the theoretical insights for three popular large-batch training techniques.
arXiv Detail & Related papers (2020-02-04T23:03:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.