Neural Generalized Ordinary Differential Equations with Layer-varying Parameters

• URL: http://arxiv.org/abs/2209.10633v1
• Date: Wed, 21 Sep 2022 20:02:28 GMT
• Title: Neural Generalized Ordinary Differential Equations with Layer-varying Parameters
• Authors: Duo Yu, Hongyu Miao, Hulin Wu
• Abstract summary: We show that the layer-varying Neural-GODE is more flexible and general than the standard Neural-ODE. The Neural-GODE enjoys the computational and memory benefits while performing comparably to ResNets in prediction accuracy.
• Score: 1.3691539554014036
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deep residual networks (ResNets) have shown state-of-the-art performance in various real-world applications. Recently, the ResNets model was reparameterized and interpreted as solutions to a continuous ordinary differential equation or Neural-ODE model. In this study, we propose a neural generalized ordinary differential equation (Neural-GODE) model with layer-varying parameters to further extend the Neural-ODE to approximate the discrete ResNets. Specifically, we use nonparametric B-spline functions to parameterize the Neural-GODE so that the trade-off between the model complexity and computational efficiency can be easily balanced. It is demonstrated that ResNets and Neural-ODE models are special cases of the proposed Neural-GODE model. Based on two benchmark datasets, MNIST and CIFAR-10, we show that the layer-varying Neural-GODE is more flexible and general than the standard Neural-ODE. Furthermore, the Neural-GODE enjoys the computational and memory benefits while performing comparably to ResNets in prediction accuracy.

Related papers

• Neural Fractional Differential Equations [2.812395851874055]
  Fractional Differential Equations (FDEs) are essential tools for modelling complex systems in science and engineering. We propose the Neural FDE, a novel deep neural network architecture that adjusts a FDE to the dynamics of data.
  arXiv  Detail & Related papers  (2024-03-05T07:45:29Z)
• Neural Laplace: Learning diverse classes of differential equations in the Laplace domain [86.52703093858631]
  We propose a unified framework for learning diverse classes of differential equations (DEs) including all the aforementioned ones. Instead of modelling the dynamics in the time domain, we model it in the Laplace domain, where the history-dependencies and discontinuities in time can be represented as summations of complex exponentials. In the experiments, Neural Laplace shows superior performance in modelling and extrapolating the trajectories of diverse classes of DEs.
  arXiv  Detail & Related papers  (2022-06-10T02:14:59Z)
• Realization Theory Of Recurrent Neural ODEs Using Polynomial System Embeddings [0.802904964931021]
  We show that neural ODE analogs of recurrent (RNN) and LongTerm Memory (ODE-LSTM) networks can be algorithmically embedded into the class of systems. This embedding input-output behavior and can be extended to other DE-LSTM architectures. We then use realization theory of systems to provide necessary conditions for an input-output to be realizable by an ODE-LSTM and sufficient conditions for minimality of such systems.
  arXiv  Detail & Related papers  (2022-05-24T11:36:18Z)
• Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEs [70.51212431290611]
  Partial differential equations (SPDEs) are significant tools for modeling dynamics in many areas including atmospheric sciences and physics. We propose the Neural Operator with Regularity Structure (NORS) which incorporates the feature vectors for modeling dynamics driven by SPDEs. We conduct experiments on various of SPDEs including the dynamic Phi41 model and the 2d Navier-Stokes equation.
  arXiv  Detail & Related papers  (2022-04-13T08:53:41Z)
• Neural Stochastic Partial Differential Equations [1.2183405753834562]
  We introduce the Neural SPDE model providing an extension to two important classes of physics-inspired neural architectures. On the one hand, it extends all the popular neural -- ordinary, controlled, rough -- differential equation models in that it is capable of processing incoming information. On the other hand, it extends Neural Operators -- recent generalizations of neural networks modelling mappings between functional spaces -- in that it can be used to learn complex SPDE solution operators.
  arXiv  Detail & Related papers  (2021-10-19T20:35:37Z)
• Sparse Flows: Pruning Continuous-depth Models [107.98191032466544]
  We show that pruning improves generalization for neural ODEs in generative modeling. We also show that pruning finds minimal and efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy.
  arXiv  Detail & Related papers  (2021-06-24T01:40:17Z)
• Accelerating Neural ODEs Using Model Order Reduction [0.0]
  We show that mathematical model order reduction methods can be used for compressing and accelerating Neural ODEs. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and operations as layers of the neural network.
  arXiv  Detail & Related papers  (2021-05-28T19:27:09Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• Neural Ordinary Differential Equation based Recurrent Neural Network Model [0.7233897166339269]
  differential equations are a promising new member in the neural network family. This paper explores the strength of the ordinary differential equation (ODE) is explored with a new extension. Two new ODE-based RNN models (GRU-ODE model and LSTM-ODE) can compute the hidden state and cell state at any point of time using an ODE solver. Experiments show that these new ODE based RNN models require less training time than Latent ODEs and conventional Neural ODEs.
  arXiv  Detail & Related papers  (2020-05-20T01:02:29Z)
• Time Dependence in Non-Autonomous Neural ODEs [74.78386661760662]
  We propose a novel family of Neural ODEs with time-varying weights. We outperform previous Neural ODE variants in both speed and representational capacity.
  arXiv  Detail & Related papers  (2020-05-05T01:41:46Z)
• Stochasticity in Neural ODEs: An Empirical Study [68.8204255655161]
  Regularization of neural networks (e.g. dropout) is a widespread technique in deep learning that allows for better generalization. We show that data augmentation during the training improves the performance of both deterministic and versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical regularization gains, making the performance of neural ODE and neural SDE negligible.
  arXiv  Detail & Related papers  (2020-02-22T22:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.