Learning Differential Equations that are Easy to Solve

• URL: http://arxiv.org/abs/2007.04504v2
• Date: Thu, 22 Oct 2020 18:56:41 GMT
• Title: Learning Differential Equations that are Easy to Solve
• Authors: Jacob Kelly, Jesse Bettencourt, Matthew James Johnson, David Duvenaud
• Abstract summary: We introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.
• Score: 26.05208133659686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Differential equations parameterized by neural networks become expensive to solve numerically as training progresses. We propose a remedy that encourages learned dynamics to be easier to solve. Specifically, we introduce a differentiable surrogate for the time cost of standard numerical solvers, using higher-order derivatives of solution trajectories. These derivatives are efficient to compute with Taylor-mode automatic differentiation. Optimizing this additional objective trades model performance against the time cost of solving the learned dynamics. We demonstrate our approach by training substantially faster, while nearly as accurate, models in supervised classification, density estimation, and time-series modelling tasks.

Related papers

• Tensor-Valued Time and Inference Path Optimization in Differential Equation-Based Generative Modeling [16.874769609089764]
  This work introduces, for the first time, a tensor-valued time that expands the conventional scalar-valued time into multiple dimensions. We also propose a novel path optimization problem designed to adaptively determine multidimensional inference trajectories.
  arXiv  Detail & Related papers  (2024-04-22T13:20:01Z)
• Faster Training of Neural ODEs Using Gau{\ss}-Legendre Quadrature [68.9206193762751]
  We propose an alternative way to speed up the training of neural ODEs. We use Gauss-Legendre quadrature to solve integrals faster than ODE-based methods. We also extend the idea to training SDEs using the Wong-Zakai theorem, by training a corresponding ODE and transferring the parameters.
  arXiv  Detail & Related papers  (2023-08-21T11:31:15Z)
• Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed! [3.222802562733787]
  Implicit layer deep learning techniques, like Neural Differential Equations, have become an important modeling framework. We develop two sampling strategies to trade off between performance and training time. Our method reduces the number of function evaluations to 0.556-0.733x and accelerates predictions by 1.3-2x.
  arXiv  Detail & Related papers  (2023-03-03T23:31:15Z)
• Physics-guided Data Augmentation for Learning the Solution Operator of Linear Differential Equations [2.1850269949775663]
  We propose a physics-guided data augmentation (PGDA) method to improve the accuracy and generalization of neural operator models. We demonstrate the advantage of PGDA on a variety of linear differential equations, showing that PGDA can improve the sample complexity and is robust to distributional shift.
  arXiv  Detail & Related papers  (2022-12-08T06:29:15Z)
• Discovering ordinary differential equations that govern time-series [65.07437364102931]
  We propose a transformer-based sequence-to-sequence model that recovers scalar autonomous ordinary differential equations (ODEs) in symbolic form from time-series data of a single observed solution of the ODE. Our method is efficiently scalable: after one-time pretraining on a large set of ODEs, we can infer the governing laws of a new observed solution in a few forward passes of the model.
  arXiv  Detail & Related papers  (2022-11-05T07:07:58Z)
• Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations [62.81701992551728]
  We present a physics-informed framework for solving time-dependent partial differential equations. Our model utilizes discrete cosine transforms to encode spatial and recurrent neural networks. We show experimental results on the Taylor-Green vortex solution to the Navier-Stokes equations.
  arXiv  Detail & Related papers  (2022-02-24T20:46:52Z)
• Taylor-Lagrange Neural Ordinary Differential Equations: Toward Fast Training and Evaluation of Neural ODEs [22.976119802895017]
  We propose a data-driven approach to the training of neural ordinary differential equations (NODEs) The proposed approach achieves the same accuracy as adaptive step-size schemes while employing only low-order Taylor expansions. A suite of numerical experiments demonstrate that TL-NODEs can be trained more than an order of magnitude faster than state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-14T23:56:19Z)
• Opening the Blackbox: Accelerating Neural Differential Equations by Regularizing Internal Solver Heuristics [0.0]
  We describe a novel regularization method that uses the internal cost of adaptive differential equation solvers combined with discrete sensitivities to guide the training process. This approach opens up the blackbox numerical analysis behind the differential equation solver's algorithm and uses its local error estimates and stiffnesss as cheap and accurate cost estimates. We demonstrate how our approach can halve the prediction time and showcases how this can increase the training time by an order of magnitude.
  arXiv  Detail & Related papers  (2021-05-09T12:03:03Z)
• Efficient Learning of Generative Models via Finite-Difference Score Matching [111.55998083406134]
  We present a generic strategy to efficiently approximate any-order directional derivative with finite difference. Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations.
  arXiv  Detail & Related papers  (2020-07-07T10:05:01Z)
• STEER: Simple Temporal Regularization For Neural ODEs [80.80350769936383]
  We propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models.
  arXiv  Detail & Related papers  (2020-06-18T17:44:50Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.