Eigen-informed NeuralODEs: Dealing with stability and convergence issues of NeuralODEs

• URL: http://arxiv.org/abs/2302.10892v1
• Date: Tue, 7 Feb 2023 14:45:39 GMT
• Title: Eigen-informed NeuralODEs: Dealing with stability and convergence issues of NeuralODEs
• Authors: Tobias Thummerer, Lars Mikelsons
• Abstract summary: We present a technique to add knowledge of ODE properties based on eigenvalues to the training objective of a NeuralODE. We show, that the presented training process is far more robust against local minima, instabilities and sparse data samples and improves training convergence and performance.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Using vanilla NeuralODEs to model large and/or complex systems often fails due two reasons: Stability and convergence. NeuralODEs are capable of describing stable as well as instable dynamic systems. Selecting an appropriate numerical solver is not trivial, because NeuralODE properties change during training. If the NeuralODE becomes more stiff, a suboptimal solver may need to perform very small solver steps, which significantly slows down the training process. If the NeuralODE becomes to instable, the numerical solver might not be able to solve it at all, which causes the training process to terminate. Often, this is tackled by choosing a computational expensive solver that is robust to instable and stiff ODEs, but at the cost of a significantly decreased training performance. Our method on the other hand, allows to enforce ODE properties that fit a specific solver or application-related boundary conditions. Concerning the convergence behavior, NeuralODEs often tend to run into local minima, especially if the system to be learned is highly dynamic and/or oscillating over multiple periods. Because of the vanishing gradient at a local minimum, the NeuralODE is often not capable of leaving it and converge to the right solution. We present a technique to add knowledge of ODE properties based on eigenvalues - like (partly) stability, oscillation capability, frequency, damping and/or stiffness - to the training objective of a NeuralODE. We exemplify our method at a linear as well as a nonlinear system model and show, that the presented training process is far more robust against local minima, instabilities and sparse data samples and improves training convergence and performance.

Related papers

• Efficient, Accurate and Stable Gradients for Neural ODEs [3.79830302036482]
  We present a class of algebraically reversible solvers that are both high-order and numerically stable. This construction naturally extends to numerical schemes for Neural CDEs and SDEs.
  arXiv  Detail & Related papers  (2024-10-15T14:36:05Z)
• On Tuning Neural ODE for Stability, Consistency and Faster Convergence [0.0]
  We propose a first-order Nesterov's accelerated gradient (NAG) based ODE-solver which is proven to be tuned vis-a-vis CCS conditions. We empirically demonstrate the efficacy of our approach by training faster, while achieving better or comparable performance against neural-ode.
  arXiv  Detail & Related papers  (2023-12-04T06:18:10Z)
• Faster Training of Neural ODEs Using Gau{\ss}-Legendre Quadrature [68.9206193762751]
  We propose an alternative way to speed up the training of neural ODEs. We use Gauss-Legendre quadrature to solve integrals faster than ODE-based methods. We also extend the idea to training SDEs using the Wong-Zakai theorem, by training a corresponding ODE and transferring the parameters.
  arXiv  Detail & Related papers  (2023-08-21T11:31:15Z)
• Experimental study of Neural ODE training with adaptive solver for dynamical systems modeling [72.84259710412293]
  Some ODE solvers called adaptive can adapt their evaluation strategy depending on the complexity of the problem at hand. This paper describes a simple set of experiments to show why adaptive solvers cannot be seamlessly leveraged as a black-box for dynamical systems modelling.
  arXiv  Detail & Related papers  (2022-11-13T17:48:04Z)
• Neural Laplace: Learning diverse classes of differential equations in the Laplace domain [86.52703093858631]
  We propose a unified framework for learning diverse classes of differential equations (DEs) including all the aforementioned ones. Instead of modelling the dynamics in the time domain, we model it in the Laplace domain, where the history-dependencies and discontinuities in time can be represented as summations of complex exponentials. In the experiments, Neural Laplace shows superior performance in modelling and extrapolating the trajectories of diverse classes of DEs.
  arXiv  Detail & Related papers  (2022-06-10T02:14:59Z)
• On the balance between the training time and interpretability of neural ODE for time series modelling [77.34726150561087]
  The paper shows that modern neural ODE cannot be reduced to simpler models for time-series modelling applications. The complexity of neural ODE is compared to or exceeds the conventional time-series modelling tools. We propose a new view on time-series modelling using combined neural networks and an ODE system approach.
  arXiv  Detail & Related papers  (2022-06-07T13:49:40Z)
• Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs [16.516974867571175]
  This paper considers learning neural ODEs using implicit ODE solvers of different orders leveraging proximal operators. The proximal implicit solver guarantees superiority over explicit solvers in numerical stability and computational efficiency.
  arXiv  Detail & Related papers  (2022-04-19T02:55:10Z)
• Neural ODE Processes [64.10282200111983]
  We introduce Neural ODE Processes (NDPs), a new class of processes determined by a distribution over Neural ODEs. We show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.
  arXiv  Detail & Related papers  (2021-03-23T09:32:06Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• ResNet After All? Neural ODEs and Their Numerical Solution [28.954378025052925]
  We show that trained Neural Ordinary Differential Equation models actually depend on the specific numerical method used during training. We propose a method that monitors the behavior of the ODE solver during training to adapt its step size.
  arXiv  Detail & Related papers  (2020-07-30T11:24:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.