A Passivity-Based Method for Accelerated Convex Optimisation

• URL: http://arxiv.org/abs/2306.11474v2
• Date: Fri, 13 Sep 2024 12:17:14 GMT
• Title: A Passivity-Based Method for Accelerated Convex Optimisation
• Abstract summary: This study presents a constructive methodology for designing accelerated convex optimisation algorithms in continuous-time domain. The two key enablers are the classical concept of passivity in control theory and the time-dependent change of variables.
• Score: 0.8747606955991705
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study presents a constructive methodology for designing accelerated convex optimisation algorithms in continuous-time domain. The two key enablers are the classical concept of passivity in control theory and the time-dependent change of variables that maps the output of the internal dynamic system to the optimisation variables. The Lyapunov function associated with the optimisation dynamics is obtained as a natural consequence of specifying the internal dynamics that drives the state evolution as a passive linear time-invariant system. The passivity-based methodology provides a general framework that has the flexibility to generate convex optimisation algorithms with the guarantee of different convergence rate bounds on the objective function value. The same principle applies to the design of online parameter update algorithms for adaptive control by re-defining the output of internal dynamics to allow for the feedback interconnection with tracking error dynamics.

Related papers

• When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay [6.625270757109195]
  We show that a single quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect.<n>This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes.
  arXiv  Detail & Related papers  (2026-09-08T17:46:17Z)
• Universal Smoothness via Bernstein Polynomials: A Constructive Approximation Approach for Activation Functions [16.856453018275467]
  The efficacy of deep neural networks is heavily reliant on the design of non-linear activation functions.<n>The proposed method guarantees strictly continuous differentiability and a non-expansive Lipschitz constant of one.<n>This approach consistently outperforms state-of-the-art baselines on standard image classification benchmarks.
  arXiv  Detail & Related papers  (2026-05-04T13:38:16Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Riemannian Lyapunov Optimizer: A Unified Framework for Optimization [6.493476506951333]
  We introduce a family of optimization algorithms that unifies classic algorithms within one geometric framework.<n>RLOs bridge control theory and modern machine learning optimization, providing a unified language and a systematic toolkit.
  arXiv  Detail & Related papers  (2026-01-29T20:00:25Z)
• TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Design [7.264986493460248]
  TIDE is a Tuning-Integrated Dynamic Evolution framework designed to decouple structural reasoning from parameter optimization.<n> experiments across nine optimization problems demonstrate that TIDE significantly outperforms state-of-the-art tuning methods.
  arXiv  Detail & Related papers  (2026-01-29T04:00:02Z)
• Optimal Control Theoretic Neural Optimizer: From Backpropagation to Dynamic Programming [29.911180907218053]
  This paper focuses on an algorithmic perspective on optimization of deep neural networks (DNNs)<n>Our motivated observation is the striking algorithmic resemblance between the Backpropagation algorithm for computing gradients in DNNs and the optimality conditions for dynamical systems.<n>The resulting physics, termed Optimal Control The Neural-theoretic (OCNOpt) enables rich algorithmic opportunities.
  arXiv  Detail & Related papers  (2025-10-15T23:39:51Z)
• Efficient Transformed Gaussian Process State-Space Models for Non-Stationary High-Dimensional Dynamical Systems [49.819436680336786]
  We propose an efficient transformed Gaussian process state-space model (ETGPSSM) for scalable and flexible modeling of high-dimensional, non-stationary dynamical systems. Specifically, our ETGPSSM integrates a single shared GP with input-dependent normalizing flows, yielding an expressive implicit process prior that captures complex, non-stationary transition dynamics. Our ETGPSSM outperforms existing GPSSMs and neural network-based SSMs in terms of computational efficiency and accuracy.
  arXiv  Detail & Related papers  (2025-03-24T03:19:45Z)
• Blockwise Optimization for Projective Variational Quantum Dynamics (BLOP-VQD): Algorithm and Implementation for Lattice Systems [0.0]
  We present an efficient approach to simulate real-time quantum dynamics using Projected Variational Quantum Dynamics. Our method selectively optimize one block at a time while keeping the others fixed, allowing for significant reductions in computational overhead. We demonstrate the performance of the proposed methods in a series of spin-lattice models with varying sizes and complexity.
  arXiv  Detail & Related papers  (2025-03-24T01:48:37Z)
• Tensor-Valued Time and Inference Path Optimization in Differential Equation-Based Generative Modeling [16.874769609089764]
  This work introduces, for the first time, a tensor-valued time that expands the conventional scalar-valued time into multiple dimensions. We also propose a novel path optimization problem designed to adaptively determine multidimensional inference trajectories.
  arXiv  Detail & Related papers  (2024-04-22T13:20:01Z)
• Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization [50.38518771642365]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. A central challenge in this setting is backpropagation through the solution of an optimization problem, which often lacks a closed form. This paper provides theoretical insights into the backward pass of unrolled optimization, showing that it is equivalent to the solution of a linear system by a particular iterative method. A system called Folded Optimization is proposed to construct more efficient backpropagation rules from unrolled solver implementations.
  arXiv  Detail & Related papers  (2023-12-28T23:15:18Z)
• Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
  arXiv  Detail & Related papers  (2023-01-28T01:50:42Z)
• Neural ODEs as Feedback Policies for Nonlinear Optimal Control [1.8514606155611764]
  We use Neural ordinary differential equations (Neural ODEs) to model continuous time dynamics as differential equations parametrized with neural networks. We propose the use of a neural control policy posed as a Neural ODE to solve general nonlinear optimal control problems.
  arXiv  Detail & Related papers  (2022-10-20T13:19:26Z)
• Adaptive Online Optimization with Predictions: Static and Dynamic Environments [5.553963083111226]
  We propose new step-size rules and OCO algorithms that exploit gradient predictions, function predictions and dynamics. The proposed algorithms enjoy static and dynamic regret bounds in terms of the dynamics of the reference action sequence. We present results for both convex and strongly convex costs.
  arXiv  Detail & Related papers  (2022-05-01T11:03:33Z)
• Optimisation of Structured Neural Controller Based on Continuous-Time Policy Gradient [2.297079626504224]
  This study presents a policy optimisation framework for structured nonlinear control of continuous-time (deterministic) dynamic systems. The proposed approach prescribes a structure for the controller based on relevant scientific knowledge. Numerical experiments on aerospace applications illustrate the utility of the structured nonlinear controller optimisation framework.
  arXiv  Detail & Related papers  (2022-01-17T08:06:19Z)
• Breaking the Convergence Barrier: Optimization via Fixed-Time Convergent Flows [4.817429789586127]
  We introduce a Poly-based optimization framework for achieving acceleration, based on the notion of fixed-time stability dynamical systems. We validate the accelerated convergence properties of the proposed schemes on a range of numerical examples against the state-of-the-art optimization algorithms.
  arXiv  Detail & Related papers  (2021-12-02T16:04:40Z)
• Robust Value Iteration for Continuous Control Tasks [99.00362538261972]
  When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well. We present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain. We show that robust value is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.
  arXiv  Detail & Related papers  (2021-05-25T19:48:35Z)
• Value Iteration in Continuous Actions, States and Time [99.00362538261972]
  We propose a continuous fitted value iteration (cFVI) algorithm for continuous states and actions. The optimal policy can be derived for non-linear control-affine dynamics. Videos of the physical system are available at urlhttps://sites.google.com/view/value-iteration.
  arXiv  Detail & Related papers  (2021-05-10T21:40:56Z)
• EOS: a Parallel, Self-Adaptive, Multi-Population Evolutionary Algorithm for Constrained Global Optimization [68.8204255655161]
  EOS is a global optimization algorithm for constrained and unconstrained problems of real-valued variables. It implements a number of improvements to the well-known Differential Evolution (DE) algorithm. Results prove that EOSis capable of achieving increased performance compared to state-of-the-art single-population self-adaptive DE algorithms.
  arXiv  Detail & Related papers  (2020-07-09T10:19:22Z)
• Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives [97.16266088683061]
  The article rigorously establishes why symplectic discretization schemes are important for momentum-based optimization algorithms. It provides a characterization of algorithms that exhibit accelerated convergence.
  arXiv  Detail & Related papers  (2020-02-28T00:32:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.