Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods

• URL: http://arxiv.org/abs/2412.01181v1
• Date: Mon, 02 Dec 2024 06:40:08 GMT
• Title: Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods
• Authors: Colby Fronk, Linda Petzold,
• Abstract summary: Stiff ordinary differential equations (ODEs) are common in many science and engineering fields. Standard neural ODE approaches struggle to accurately learn stiff systems. This paper expands on our earlier work by exploring explicit exponential integration methods.
• Score: 3.941173292703699
• License:
• Abstract: Stiff ordinary differential equations (ODEs) are common in many science and engineering fields, but standard neural ODE approaches struggle to accurately learn these stiff systems, posing a significant barrier to widespread adoption of neural ODEs. In our earlier work, we addressed this challenge by utilizing single-step implicit methods for solving stiff neural ODEs. While effective, these implicit methods are computationally costly and can be complex to implement. This paper expands on our earlier work by exploring explicit exponential integration methods as a more efficient alternative. We evaluate the potential of these explicit methods to handle stiff dynamics in neural ODEs, aiming to enhance their applicability to a broader range of scientific and engineering problems. We found the integrating factor Euler (IF Euler) method to excel in stability and efficiency. While implicit schemes failed to train the stiff Van der Pol oscillator, the IF Euler method succeeded, even with large step sizes. However, IF Euler's first-order accuracy limits its use, leaving the development of higher-order methods for stiff neural ODEs an open research problem.

Related papers

• Stiff Transfer Learning for Physics-Informed Neural Networks [1.5361702135159845]
  We propose a novel approach, stiff transfer learning for physics-informed neural networks (STL-PINNs) to tackle stiff ordinary differential equations (ODEs) and partial differential equations (PDEs) Our methodology involves training a Multi-Head-PINN in a low-stiff regime, and obtaining the final solution in a high stiff regime by transfer learning. This addresses the failure modes related to stiffness in PINNs while maintaining computational efficiency by computing "one-shot" solutions.
  arXiv  Detail & Related papers  (2025-01-28T20:27:38Z)
• Semi-Implicit Neural Ordinary Differential Equations [5.196303789025002]
  We present a semi-implicit neural ODE approach that exploits the partitionable structure of the underlying dynamics. Our technique leads to an implicit neural network with significant computational advantages over existing approaches.
  arXiv  Detail & Related papers  (2024-12-15T20:21:02Z)
• Training Stiff Neural Ordinary Differential Equations with Implicit Single-Step Methods [3.941173292703699]
  Stiff systems of ordinary differential equations (ODEs) are pervasive in many science and engineering fields. Standard neural ODE approaches struggle to learn them. This paper proposes an approach based on single-step implicit schemes to enable neural ODEs to handle stiffness.
  arXiv  Detail & Related papers  (2024-10-08T01:08:17Z)
• Faster Training of Neural ODEs Using Gau{\ss}-Legendre Quadrature [68.9206193762751]
  We propose an alternative way to speed up the training of neural ODEs. We use Gauss-Legendre quadrature to solve integrals faster than ODE-based methods. We also extend the idea to training SDEs using the Wong-Zakai theorem, by training a corresponding ODE and transferring the parameters.
  arXiv  Detail & Related papers  (2023-08-21T11:31:15Z)
• Accelerated primal-dual methods with enlarged step sizes and operator learning for nonsmooth optimal control problems [3.1006429989273063]
  We focus on the application of a primal-dual method, with which different types of variables can be treated individually. For the accelerated primal-dual method with larger step sizes, its convergence can be proved rigorously while it numerically accelerates the original primal-dual method. For the operator learning acceleration, we construct deep neural network surrogate models for the involved PDEs.
  arXiv  Detail & Related papers  (2023-07-01T10:39:07Z)
• Proximal Implicit ODE Solvers for Accelerating Learning Neural ODEs [16.516974867571175]
  This paper considers learning neural ODEs using implicit ODE solvers of different orders leveraging proximal operators. The proximal implicit solver guarantees superiority over explicit solvers in numerical stability and computational efficiency.
  arXiv  Detail & Related papers  (2022-04-19T02:55:10Z)
• Message Passing Neural PDE Solvers [60.77761603258397]
  We build a neural message passing solver, replacing allally designed components in the graph with backprop-optimized neural function approximators. We show that neural message passing solvers representationally contain some classical methods, such as finite differences, finite volumes, and WENO schemes. We validate our method on various fluid-like flow problems, demonstrating fast, stable, and accurate performance across different domain topologies, equation parameters, discretizations, etc., in 1D and 2D.
  arXiv  Detail & Related papers  (2022-02-07T17:47:46Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• STEER: Simple Temporal Regularization For Neural ODEs [80.80350769936383]
  We propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models.
  arXiv  Detail & Related papers  (2020-06-18T17:44:50Z)
• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs [71.26657499537366]
  We propose a simple literature-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method to train neural ODEs on classification, density estimation, and inference approximation tasks.
  arXiv  Detail & Related papers  (2020-03-11T13:15:57Z)
• Stochasticity in Neural ODEs: An Empirical Study [68.8204255655161]
  Regularization of neural networks (e.g. dropout) is a widespread technique in deep learning that allows for better generalization. We show that data augmentation during the training improves the performance of both deterministic and versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical regularization gains, making the performance of neural ODE and neural SDE negligible.
  arXiv  Detail & Related papers  (2020-02-22T22:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.