Differentiable Multiple Shooting Layers

• URL: http://arxiv.org/abs/2106.03885v1
• Date: Mon, 7 Jun 2021 18:05:44 GMT
• Title: Differentiable Multiple Shooting Layers
• Authors: Stefano Massaroli, Michael Poli, Sho Sonoda, Taji Suzuki, Jinkyoo Park, Atsushi Yamashita and Hajime Asama
• Abstract summary: Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. We investigate the speedups obtained through application of MSL inference in neural controlled differential equations (Neural CDEs) for time series classification of medical data.
• Score: 18.37758865401204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We detail a novel class of implicit neural models. Leveraging time-parallel methods for differential equations, Multiple Shooting Layers (MSLs) seek solutions of initial value problems via parallelizable root-finding algorithms. MSLs broadly serve as drop-in replacements for neural ordinary differential equations (Neural ODEs) with improved efficiency in number of function evaluations (NFEs) and wall-clock inference time. We develop the algorithmic framework of MSLs, analyzing the different choices of solution methods from a theoretical and computational perspective. MSLs are showcased in long horizon optimal control of ODEs and PDEs and as latent models for sequence generation. Finally, we investigate the speedups obtained through application of MSL inference in neural controlled differential equations (Neural CDEs) for time series classification of medical data.

Related papers

• Score-based Neural Ordinary Differential Equations for Computing Mean Field Control Problems [13.285775352653546]
  This paper proposes a system of neural differential equations representing first- and second-order score functions along trajectories based on deep neural networks. We reformulate the mean viscous field control (MFC) problem with individual noises into an unconstrained optimization problem framed by the proposed neural ODE system.
  arXiv  Detail & Related papers  (2024-09-24T21:45:55Z)
• A Deep Neural Network Framework for Solving Forward and Inverse Problems in Delay Differential Equations [12.888147363070749]
  We propose a unified framework for delay differential equations (DDEs) based on deep neural networks (DNNs) This framework could embed delay differential equations into neural networks to accommodate the diverse requirements of DDEs. In addressing inverse problems, the NDDE framework can utilize observational data to perform precise estimation of single or multiple delay parameters.
  arXiv  Detail & Related papers  (2024-08-17T13:41:34Z)
• Solving partial differential equations with sampled neural networks [1.8590821261905535]
  Approximation of solutions to partial differential equations (PDE) is an important problem in computational science and engineering. We discuss how sampling the hidden weights and biases of the ansatz network from data-agnostic and data-dependent probability distributions allows us to progress on both challenges.
  arXiv  Detail & Related papers  (2024-05-31T14:24:39Z)
• Neural Spectral Methods: Self-supervised learning in the spectral domain [0.0]
  We present Neural Spectral Methods, a technique to solve parametric Partial Equations (PDEs) Our method uses bases to learn PDE solutions as mappings between spectral coefficients. Our experimental results demonstrate that our method significantly outperforms previous machine learning approaches terms of speed and accuracy.
  arXiv  Detail & Related papers  (2023-12-08T18:20:43Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• Locally Regularized Neural Differential Equations: Some Black Boxes Were Meant to Remain Closed! [3.222802562733787]
  Implicit layer deep learning techniques, like Neural Differential Equations, have become an important modeling framework. We develop two sampling strategies to trade off between performance and training time. Our method reduces the number of function evaluations to 0.556-0.733x and accelerates predictions by 1.3-2x.
  arXiv  Detail & Related papers  (2023-03-03T23:31:15Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• Neural Laplace: Learning diverse classes of differential equations in the Laplace domain [86.52703093858631]
  We propose a unified framework for learning diverse classes of differential equations (DEs) including all the aforementioned ones. Instead of modelling the dynamics in the time domain, we model it in the Laplace domain, where the history-dependencies and discontinuities in time can be represented as summations of complex exponentials. In the experiments, Neural Laplace shows superior performance in modelling and extrapolating the trajectories of diverse classes of DEs.
  arXiv  Detail & Related papers  (2022-06-10T02:14:59Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• Large-scale Neural Solvers for Partial Differential Equations [48.7576911714538]
  Solving partial differential equations (PDE) is an indispensable part of many branches of science as many processes can be modelled in terms of PDEs. Recent numerical solvers require manual discretization of the underlying equation as well as sophisticated, tailored code for distributed computing. We examine the applicability of continuous, mesh-free neural solvers for partial differential equations, physics-informed neural networks (PINNs) We discuss the accuracy of GatedPINN with respect to analytical solutions -- as well as state-of-the-art numerical solvers, such as spectral solvers.
  arXiv  Detail & Related papers  (2020-09-08T13:26:51Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.