Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!

• URL: http://arxiv.org/abs/2303.02262v3
• Date: Fri, 2 Jun 2023 14:52:33 GMT
• Title: Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed!
• Authors: Avik Pal, Alan Edelman, Chris Rackauckas
• Abstract summary: Implicit layer deep learning techniques, like Neural Differential Equations, have become an important modeling framework. We develop two sampling strategies to trade off between performance and training time. Our method reduces the number of function evaluations to 0.556-0.733x and accelerates predictions by 1.3-2x.
• Score: 3.222802562733787
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Implicit layer deep learning techniques, like Neural Differential Equations, have become an important modeling framework due to their ability to adapt to new problems automatically. Training a neural differential equation is effectively a search over a space of plausible dynamical systems. However, controlling the computational cost for these models is difficult since it relies on the number of steps the adaptive solver takes. Most prior works have used higher-order methods to reduce prediction timings while greatly increasing training time or reducing both training and prediction timings by relying on specific training algorithms, which are harder to use as a drop-in replacement due to strict requirements on automatic differentiation. In this manuscript, we use internal cost heuristics of adaptive differential equation solvers at stochastic time points to guide the training toward learning a dynamical system that is easier to integrate. We "close the black-box" and allow the use of our method with any adjoint technique for gradient calculations of the differential equation solution. We perform experimental studies to compare our method to global regularization to show that we attain similar performance numbers without compromising the flexibility of implementation on ordinary differential equations (ODEs) and stochastic differential equations (SDEs). We develop two sampling strategies to trade off between performance and training time. Our method reduces the number of function evaluations to 0.556-0.733x and accelerates predictions by 1.3-2x.

Related papers

• PICL: Physics Informed Contrastive Learning for Partial Differential Equations [7.136205674624813]
  We develop a novel contrastive pretraining framework that improves neural operator generalization across multiple governing equations simultaneously. A combination of physics-informed system evolution and latent-space model output are anchored to input data and used in our distance function. We find that physics-informed contrastive pretraining improves accuracy for the Fourier Neural Operator in fixed-future and autoregressive rollout tasks for the 1D and 2D Heat, Burgers', and linear advection equations.
  arXiv  Detail & Related papers  (2024-01-29T17:32:22Z)
• Solving differential equations using physics informed deep learning: a hand-on tutorial with benchmark tests [0.0]
  We revisit the original approach of using deep learning and neural networks to solve differential equations. We focus on the possibility to use the least possible amount of data into the training process. A tutorial on a simple equation model illustrates how to put into practice the method for ordinary differential equations.
  arXiv  Detail & Related papers  (2023-02-23T16:08:39Z)
• Experimental study of Neural ODE training with adaptive solver for dynamical systems modeling [72.84259710412293]
  Some ODE solvers called adaptive can adapt their evaluation strategy depending on the complexity of the problem at hand. This paper describes a simple set of experiments to show why adaptive solvers cannot be seamlessly leveraged as a black-box for dynamical systems modelling.
  arXiv  Detail & Related papers  (2022-11-13T17:48:04Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• Efficient Differentiable Simulation of Articulated Bodies [89.64118042429287]
  We present a method for efficient differentiable simulation of articulated bodies. This enables integration of articulated body dynamics into deep learning frameworks. We show that reinforcement learning with articulated systems can be accelerated using gradients provided by our method.
  arXiv  Detail & Related papers  (2021-09-16T04:48:13Z)
• Distributional Gradient Matching for Learning Uncertain Neural Dynamics Models [38.17499046781131]
  We propose a novel approach towards estimating uncertain neural ODEs, avoiding the numerical integration bottleneck. Our algorithm - distributional gradient matching (DGM) - jointly trains a smoother and a dynamics model and matches their gradients via minimizing a Wasserstein loss. Our experiments show that, compared to traditional approximate inference methods based on numerical integration, our approach is faster to train, faster at predicting previously unseen trajectories, and in the context of neural ODEs, significantly more accurate.
  arXiv  Detail & Related papers  (2021-06-22T08:40:51Z)
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics [0.0]
  We describe a novel regularization method that uses the internal cost of adaptive differential equation solvers combined with discrete sensitivities to guide the training process. This approach opens up the blackbox numerical analysis behind the differential equation solver's algorithm and uses its local error estimates and stiffnesss as cheap and accurate cost estimates. We demonstrate how our approach can halve the prediction time and showcases how this can increase the training time by an order of magnitude.
  arXiv  Detail & Related papers  (2021-05-09T12:03:03Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• DiffPD: Differentiable Projective Dynamics with Contact [65.88720481593118]
  We present DiffPD, an efficient differentiable soft-body simulator with implicit time integration. We evaluate the performance of DiffPD and observe a speedup of 4-19 times compared to the standard Newton's method in various applications.
  arXiv  Detail & Related papers  (2021-01-15T00:13:33Z)
• Learning Differential Equations that are Easy to Solve [26.05208133659686]
  We introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.
  arXiv  Detail & Related papers  (2020-07-09T01:39:34Z)
• Physarum Powered Differentiable Linear Programming Layers and Applications [48.77235931652611]
  We propose an efficient and differentiable solver for general linear programming problems. We show the use of our solver in a video segmentation task and meta-learning for few-shot learning.
  arXiv  Detail & Related papers  (2020-04-30T01:50:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.