Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for
Optimization
- URL: http://arxiv.org/abs/2201.11137v1
- Date: Wed, 26 Jan 2022 19:00:05 GMT
- Title: Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for
Optimization
- Authors: G. Bruno De Luca and Eva Silverstein
- Abstract summary: We introduce a framework based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and its key properties analytically and numerically.
The prototype is establish a discretization of Born-Infeld dynamics, depending on the objective function.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce a novel framework for optimization based on energy-conserving
Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its
key properties analytically and numerically. The prototype is a discretization
of Born-Infeld dynamics, with a squared relativistic speed limit depending on
the objective function. This class of frictionless, energy-conserving
optimizers proceeds unobstructed until slowing naturally near the minimal loss,
which dominates the phase space volume of the system. Building from studies of
chaotic systems such as dynamical billiards, we formulate a specific algorithm
with good performance on machine learning and PDE-solving tasks, including
generalization. It cannot stop at a high local minimum and cannot overshoot the
global minimum, yielding an advantage in non-convex loss functions, and
proceeds faster than GD+momentum in shallow valleys.
Related papers
- A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning [74.80956524812714]
We tackle the general differentiable meta learning problem that is ubiquitous in modern deep learning.
These problems are often formalized as Bi-Level optimizations (BLO)
We introduce a novel perspective by turning a given BLO problem into a ii optimization, where the inner loss function becomes a smooth distribution, and the outer loss becomes an expected loss over the inner distribution.
arXiv Detail & Related papers (2024-10-14T12:10:06Z) - Convergence of mean-field Langevin dynamics: Time and space
discretization, stochastic gradient, and variance reduction [49.66486092259376]
The mean-field Langevin dynamics (MFLD) is a nonlinear generalization of the Langevin dynamics that incorporates a distribution-dependent drift.
Recent works have shown that MFLD globally minimizes an entropy-regularized convex functional in the space of measures.
We provide a framework to prove a uniform-in-time propagation of chaos for MFLD that takes into account the errors due to finite-particle approximation, time-discretization, and gradient approximation.
arXiv Detail & Related papers (2023-06-12T16:28:11Z) - Accelerated First-Order Optimization under Nonlinear Constraints [73.2273449996098]
We exploit between first-order algorithms for constrained optimization and non-smooth systems to design a new class of accelerated first-order algorithms.
An important property of these algorithms is that constraints are expressed in terms of velocities instead of sparse variables.
arXiv Detail & Related papers (2023-02-01T08:50:48Z) - GPU-accelerated Optimal Path Planning in Stochastic Dynamic Environments [0.0]
Planning time and energy optimal paths for autonomous marine vehicles is essential to reduce operational costs.
Decision Processes (MDPs) provide a natural framework for sequential decision-making for robotic agents in such environments.
We introduce an efficient end-to-end-accelerated algorithm that builds the MDP model and solves the MDP to compute an optimal policy.
arXiv Detail & Related papers (2021-09-02T12:14:34Z) - Optimization on manifolds: A symplectic approach [127.54402681305629]
We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems.
Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
arXiv Detail & Related papers (2021-07-23T13:43:34Z) - Minimax Optimization with Smooth Algorithmic Adversaries [59.47122537182611]
We propose a new algorithm for the min-player against smooth algorithms deployed by an adversary.
Our algorithm is guaranteed to make monotonic progress having no limit cycles, and to find an appropriate number of gradient ascents.
arXiv Detail & Related papers (2021-06-02T22:03:36Z) - Stochastic gradient descent with noise of machine learning type. Part I:
Discrete time analysis [0.0]
gradient descent (SGD) is one of the most popular algorithms in modern machine learning.
In this article, we discuss some of the common properties of energy landscapes and the noise encountered in machine learning problems.
arXiv Detail & Related papers (2021-05-04T17:52:20Z) - Fairly Constricted Multi-Objective Particle Swarm Optimization [0.0]
We extend the state of the art Multi-objective optimization (MOO) solver, SMPSO, by incorporating exponentially-averaged momentum (EM) in it.
The proposed solver matches the performance of SMPSO across the ZDT, DTLZ and WFG problem suites and even outperforms it in certain instances.
arXiv Detail & Related papers (2021-04-10T14:39:59Z) - Efficient Methods for Structured Nonconvex-Nonconcave Min-Max
Optimization [98.0595480384208]
We propose a generalization extraient spaces which converges to a stationary point.
The algorithm applies not only to general $p$-normed spaces, but also to general $p$-dimensional vector spaces.
arXiv Detail & Related papers (2020-10-31T21:35:42Z) - CoolMomentum: A Method for Stochastic Optimization by Langevin Dynamics
with Simulated Annealing [23.87373187143897]
Deep learning applications require global optimization of non- objective functions, which have multiple local minima.
We show that a coordinate learning algorithm can be used to resolve the same problem in physical simulations.
arXiv Detail & Related papers (2020-05-29T14:44:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.