Investigation into the Training Dynamics of Learned Optimizers
- URL: http://arxiv.org/abs/2312.07174v1
- Date: Tue, 12 Dec 2023 11:18:43 GMT
- Title: Investigation into the Training Dynamics of Learned Optimizers
- Authors: Jan Sobotka, Petr \v{S}im\'anek, Daniel Va\v{s}ata
- Abstract summary: We look at the concept of learneds as a way to accelerate the optimization process by replacing traditional, hand-crafted algorithms with meta-learned functions.
Our work examines their optimization from the perspective of network architecture symmetries and update parameters.
We identify several key insights that demonstrate how each approach can benefit from the strengths of the other.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Optimization is an integral part of modern deep learning. Recently, the
concept of learned optimizers has emerged as a way to accelerate this
optimization process by replacing traditional, hand-crafted algorithms with
meta-learned functions. Despite the initial promising results of these methods,
issues with stability and generalization still remain, limiting their practical
use. Moreover, their inner workings and behavior under different conditions are
not yet fully understood, making it difficult to come up with improvements. For
this reason, our work examines their optimization trajectories from the
perspective of network architecture symmetries and parameter update
distributions. Furthermore, by contrasting the learned optimizers with their
manually designed counterparts, we identify several key insights that
demonstrate how each approach can benefit from the strengths of the other.
Related papers
- Learning Joint Models of Prediction and Optimization [56.04498536842065]
Predict-Then-Then framework uses machine learning models to predict unknown parameters of an optimization problem from features before solving.
This paper proposes an alternative method, in which optimal solutions are learned directly from the observable features by joint predictive models.
arXiv Detail & Related papers (2024-09-07T19:52:14Z) - Narrowing the Focus: Learned Optimizers for Pretrained Models [24.685918556547055]
We propose a novel technique that learns a layer-specific linear combination of update directions provided by a set of base work tasks.
When evaluated on an image, this specialized significantly outperforms both traditional off-the-shelf methods such as Adam, as well existing general learneds.
arXiv Detail & Related papers (2024-08-17T23:55:19Z) - Predict-Then-Optimize by Proxy: Learning Joint Models of Prediction and
Optimization [59.386153202037086]
Predict-Then- framework uses machine learning models to predict unknown parameters of an optimization problem from features before solving.
This approach can be inefficient and requires handcrafted, problem-specific rules for backpropagation through the optimization step.
This paper proposes an alternative method, in which optimal solutions are learned directly from the observable features by predictive models.
arXiv Detail & Related papers (2023-11-22T01:32:06Z) - Understanding Optimization of Deep Learning via Jacobian Matrix and
Lipschitz Constant [18.592094066642364]
This article provides a comprehensive understanding of optimization in deep learning.
We focus on the challenges of gradient vanishing and gradient exploding, which normally lead to diminished model representational ability and training instability, respectively.
To help understand the current optimization methodologies, we categorize them into two classes: explicit optimization and implicit optimization.
arXiv Detail & Related papers (2023-06-15T17:59:27Z) - Learning to Optimize for Reinforcement Learning [58.01132862590378]
Reinforcement learning (RL) is essentially different from supervised learning, and in practice, these learneds do not work well even in simple RL tasks.
Agent-gradient distribution is non-independent and identically distributed, leading to inefficient meta-training.
We show that, although only trained in toy tasks, our learned can generalize unseen complex tasks in Brax.
arXiv Detail & Related papers (2023-02-03T00:11:02Z) - Learning to Optimize with Dynamic Mode Decomposition [0.0]
We show how to utilize the dynamic mode decomposition method for extracting informative features about optimization dynamics.
We show that our learned generalizes much better to unseen optimization problems in short.
arXiv Detail & Related papers (2022-11-29T14:55:59Z) - A Closer Look at Learned Optimization: Stability, Robustness, and
Inductive Biases [44.01339030872185]
Blackbox learneds often struggle with stability and generalization when applied to tasks unlike those in their meta-training set.
We investigate the inductive biases and stability properties of optimization algorithms, and apply the resulting insights to designing inductive biases for blackboxs.
We learn to a variety of neural network training tasks, where it outperforms the current state of the art learned.
arXiv Detail & Related papers (2022-09-22T17:47:21Z) - Meta Mirror Descent: Optimiser Learning for Fast Convergence [85.98034682899855]
We take a different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergence.
Within this paradigm, we formalise a novel meta-learning objective of minimising the regret bound of learning.
Unlike many meta-learned optimisers, it also supports convergence and generalisation guarantees and uniquely does so without requiring validation data.
arXiv Detail & Related papers (2022-03-05T11:41:13Z) - Reverse engineering learned optimizers reveals known and novel
mechanisms [50.50540910474342]
Learneds are algorithms that can themselves be trained to solve optimization problems.
Our results help elucidate the previously murky understanding of how learneds work, and establish tools for interpreting future learneds.
arXiv Detail & Related papers (2020-11-04T07:12:43Z) - Learning with Differentiable Perturbed Optimizers [54.351317101356614]
We propose a systematic method to transform operations into operations that are differentiable and never locally constant.
Our approach relies on perturbeds, and can be used readily together with existing solvers.
We show how this framework can be connected to a family of losses developed in structured prediction, and give theoretical guarantees for their use in learning tasks.
arXiv Detail & Related papers (2020-02-20T11:11:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.