Semi-Implicit Neural Ordinary Differential Equations

• URL: http://arxiv.org/abs/2412.11301v1
• Date: Sun, 15 Dec 2024 20:21:02 GMT
• Title: Semi-Implicit Neural Ordinary Differential Equations
• Authors: Hong Zhang, Ying Liu, Romit Maulik,
• Abstract summary: We present a semi-implicit neural ODE approach that exploits the partitionable structure of the underlying dynamics. Our technique leads to an implicit neural network with significant computational advantages over existing approaches.
• Score: 5.196303789025002
• License:
• Abstract: Classical neural ODEs trained with explicit methods are intrinsically limited by stability, crippling their efficiency and robustness for stiff learning problems that are common in graph learning and scientific machine learning. We present a semi-implicit neural ODE approach that exploits the partitionable structure of the underlying dynamics. Our technique leads to an implicit neural network with significant computational advantages over existing approaches because of enhanced stability and efficient linear solves during time integration. We show that our approach outperforms existing approaches on a variety of applications including graph classification and learning complex dynamical systems. We also demonstrate that our approach can train challenging neural ODEs where both explicit methods and fully implicit methods are intractable.

Related papers

• Training Stiff Neural Ordinary Differential Equations with Explicit Exponential Integration Methods [3.941173292703699]
  Stiff ordinary differential equations (ODEs) are common in many science and engineering fields. Standard neural ODE approaches struggle to accurately learn stiff systems. This paper expands on our earlier work by exploring explicit exponential integration methods.
  arXiv  Detail & Related papers  (2024-12-02T06:40:08Z)
• Learning Controlled Stochastic Differential Equations [61.82896036131116]
  This work proposes a novel method for estimating both drift and diffusion coefficients of continuous, multidimensional, nonlinear controlled differential equations with non-uniform diffusion. We provide strong theoretical guarantees, including finite-sample bounds for (L2), (Linfty), and risk metrics, with learning rates adaptive to coefficients' regularity. Our method is available as an open-source Python library.
  arXiv  Detail & Related papers  (2024-11-04T11:09:58Z)
• Projected Neural Differential Equations for Learning Constrained Dynamics [3.570367665112327]
  We introduce a new method for constraining neural differential equations based on projection of the learned vector field to the tangent space of the constraint manifold. PNDEs outperform existing methods while requiring fewer hyper parameters. The proposed approach demonstrates significant potential for enhancing the modeling of constrained dynamical systems.
  arXiv  Detail & Related papers  (2024-10-31T06:32:43Z)
• Training Stiff Neural Ordinary Differential Equations with Implicit Single-Step Methods [3.941173292703699]
  Stiff systems of ordinary differential equations (ODEs) are pervasive in many science and engineering fields. Standard neural ODE approaches struggle to learn them. This paper proposes an approach based on single-step implicit schemes to enable neural ODEs to handle stiffness.
  arXiv  Detail & Related papers  (2024-10-08T01:08:17Z)
• Adaptive Feedforward Gradient Estimation in Neural ODEs [0.0]
  We propose a novel approach that leverages adaptive feedforward gradient estimation to improve the efficiency, consistency, and interpretability of Neural ODEs. Our method eliminates the need for backpropagation and the adjoint method, reducing computational overhead and memory usage while maintaining accuracy.
  arXiv  Detail & Related papers  (2024-09-22T18:21:01Z)
• Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
  We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences. It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations. Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
  arXiv  Detail & Related papers  (2024-02-20T15:23:24Z)
• Stabilized Neural Differential Equations for Learning Dynamics with Explicit Constraints [4.656302602746229]
  We propose stabilized neural differential equations (SNDEs) to enforce arbitrary manifold constraints for neural differential equations. Our approach is based on a stabilization term that, when added to the original dynamics, renders the constraint manifold provably stable. Due to its simplicity, our method is compatible with all common neural differential equation (NDE) models and broadly applicable.
  arXiv  Detail & Related papers  (2023-06-16T10:16:59Z)
• IQ-Learn: Inverse soft-Q Learning for Imitation [95.06031307730245]
  imitation learning from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function.
  arXiv  Detail & Related papers  (2021-06-23T03:43:10Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Interpolation Technique to Speed Up Gradients Propagation in Neural ODEs [71.26657499537366]
  We propose a simple literature-based method for the efficient approximation of gradients in neural ODE models. We compare it with the reverse dynamic method to train neural ODEs on classification, density estimation, and inference approximation tasks.
  arXiv  Detail & Related papers  (2020-03-11T13:15:57Z)
• Stochasticity in Neural ODEs: An Empirical Study [68.8204255655161]
  Regularization of neural networks (e.g. dropout) is a widespread technique in deep learning that allows for better generalization. We show that data augmentation during the training improves the performance of both deterministic and versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical regularization gains, making the performance of neural ODE and neural SDE negligible.
  arXiv  Detail & Related papers  (2020-02-22T22:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.