Time Dependence in Non-Autonomous Neural ODEs

• URL: http://arxiv.org/abs/2005.01906v2
• Date: Wed, 6 May 2020 16:40:50 GMT
• Title: Time Dependence in Non-Autonomous Neural ODEs
• Authors: Jared Quincy Davis, Krzysztof Choromanski, Jake Varley, Honglak Lee, Jean-Jacques Slotine, Valerii Likhosterov, Adrian Weller, Ameesh Makadia, Vikas Sindhwani
• Abstract summary: We propose a novel family of Neural ODEs with time-varying weights. We outperform previous Neural ODE variants in both speed and representational capacity.
• Score: 74.78386661760662
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural Ordinary Differential Equations (ODEs) are elegant reinterpretations of deep networks where continuous time can replace the discrete notion of depth, ODE solvers perform forward propagation, and the adjoint method enables efficient, constant memory backpropagation. Neural ODEs are universal approximators only when they are non-autonomous, that is, the dynamics depends explicitly on time. We propose a novel family of Neural ODEs with time-varying weights, where time-dependence is non-parametric, and the smoothness of weight trajectories can be explicitly controlled to allow a tradeoff between expressiveness and efficiency. Using this enhanced expressiveness, we outperform previous Neural ODE variants in both speed and representational capacity, ultimately outperforming standard ResNet and CNN models on select image classification and video prediction tasks.

Related papers

• Balanced Neural ODEs: nonlinear model order reduction and Koopman operator approximations [0.0]
  Variational Autoencoders (VAEs) are a powerful framework for learning compact latent representations. NeuralODEs excel in learning transient system dynamics. This work combines the strengths of both to create fast surrogate models with adjustable complexity.
  arXiv  Detail & Related papers  (2024-10-14T05:45:52Z)
• Neural Delay Differential Equations: System Reconstruction and Image Classification [14.59919398960571]
  We propose a new class of continuous-depth neural networks with delay, named Neural Delay Differential Equations (NDDEs) Compared to NODEs, NDDEs have a stronger capacity of nonlinear representations. We achieve lower loss and higher accuracy not only for the data produced synthetically but also for the CIFAR10, a well-known image dataset.
  arXiv  Detail & Related papers  (2023-04-11T16:09:28Z)
• On the balance between the training time and interpretability of neural ODE for time series modelling [77.34726150561087]
  The paper shows that modern neural ODE cannot be reduced to simpler models for time-series modelling applications. The complexity of neural ODE is compared to or exceeds the conventional time-series modelling tools. We propose a new view on time-series modelling using combined neural networks and an ODE system approach.
  arXiv  Detail & Related papers  (2022-06-07T13:49:40Z)
• Accelerating Neural ODEs Using Model Order Reduction [0.0]
  We show that mathematical model order reduction methods can be used for compressing and accelerating Neural ODEs. We implement our novel compression method by developing Neural ODEs that integrate the necessary subspace-projection and operations as layers of the neural network.
  arXiv  Detail & Related papers  (2021-05-28T19:27:09Z)
• Neural ODE Processes [64.10282200111983]
  We introduce Neural ODE Processes (NDPs), a new class of processes determined by a distribution over Neural ODEs. We show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.
  arXiv  Detail & Related papers  (2021-03-23T09:32:06Z)
• Time-Reversal Symmetric ODE Network [138.02741983098454]
  Time-reversal symmetry is a fundamental property that frequently holds in classical and quantum mechanics. We propose a novel loss function that measures how well our ordinary differential equation (ODE) networks comply with this time-reversal symmetry. We show that, even for systems that do not possess the full time-reversal symmetry, TRS-ODENs can achieve better predictive performances over baselines.
  arXiv  Detail & Related papers  (2020-07-22T12:19:40Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)
• Stochasticity in Neural ODEs: An Empirical Study [68.8204255655161]
  Regularization of neural networks (e.g. dropout) is a widespread technique in deep learning that allows for better generalization. We show that data augmentation during the training improves the performance of both deterministic and versions of the same model. However, the improvements obtained by the data augmentation completely eliminate the empirical regularization gains, making the performance of neural ODE and neural SDE negligible.
  arXiv  Detail & Related papers  (2020-02-22T22:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.