Latent Neural ODEs with Sparse Bayesian Multiple Shooting

• URL: http://arxiv.org/abs/2210.03466v1
• Date: Fri, 7 Oct 2022 11:36:29 GMT
• Title: Latent Neural ODEs with Sparse Bayesian Multiple Shooting
• Authors: Valerii Iakovlev, Cagatay Yildiz, Markus Heinonen, Harri L\"ahdesm\"aki
• Abstract summary: Training dynamic models, such as neural ODEs, on long trajectories is a hard problem that requires using various tricks, such as trajectory splitting, to make model training work in practice. We propose a principled multiple shooting technique for neural ODEs that splits trajectories into manageable short segments, which are optimised in parallel. We demonstrate efficient and stable training, and state-of-the-art performance on multiple large-scale benchmark datasets.
• Score: 13.104556034767025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training dynamic models, such as neural ODEs, on long trajectories is a hard problem that requires using various tricks, such as trajectory splitting, to make model training work in practice. These methods are often heuristics with poor theoretical justifications, and require iterative manual tuning. We propose a principled multiple shooting technique for neural ODEs that splits the trajectories into manageable short segments, which are optimised in parallel, while ensuring probabilistic control on continuity over consecutive segments. We derive variational inference for our shooting-based latent neural ODE models and propose amortized encodings of irregularly sampled trajectories with a transformer-based recognition network with temporal attention and relative positional encoding. We demonstrate efficient and stable training, and state-of-the-art performance on multiple large-scale benchmark datasets.

Related papers

• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Diffusion-Model-Assisted Supervised Learning of Generative Models for Density Estimation [10.793646707711442]
  We present a framework for training generative models for density estimation. We use the score-based diffusion model to generate labeled data. Once the labeled data are generated, we can train a simple fully connected neural network to learn the generative model in the supervised manner.
  arXiv  Detail & Related papers  (2023-10-22T23:56:19Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Homotopy-based training of NeuralODEs for accurate dynamics discovery [0.0]
  We develop a new training method for NeuralODEs, based on synchronization and homotopy optimization. We show that synchronizing the model dynamics and the training data tames the originally irregular loss landscape. Our method achieves competitive or better training loss while often requiring less than half the number of training epochs.
  arXiv  Detail & Related papers  (2022-10-04T06:32:45Z)
• Rate Distortion Characteristic Modeling for Neural Image Compression [59.25700168404325]
  End-to-end optimization capability offers neural image compression (NIC) superior lossy compression performance. distinct models are required to be trained to reach different points in the rate-distortion (R-D) space. We make efforts to formulate the essential mathematical functions to describe the R-D behavior of NIC using deep network and statistical modeling.
  arXiv  Detail & Related papers  (2021-06-24T12:23:05Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• DyNODE: Neural Ordinary Differential Equations for Dynamics Modeling in Continuous Control [0.0]
  We present a novel approach that captures the underlying dynamics of a system by incorporating control in a neural ordinary differential equation framework. Results indicate that a simple DyNODE architecture when combined with an actor-critic reinforcement learning algorithm outperforms canonical neural networks.
  arXiv  Detail & Related papers  (2020-09-09T12:56:58Z)
• STEER: Simple Temporal Regularization For Neural ODEs [80.80350769936383]
  We propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models.
  arXiv  Detail & Related papers  (2020-06-18T17:44:50Z)
• Forecasting Sequential Data using Consistent Koopman Autoencoders [52.209416711500005]
  A new class of physics-based methods related to Koopman theory has been introduced, offering an alternative for processing nonlinear dynamical systems. We propose a novel Consistent Koopman Autoencoder model which, unlike the majority of existing work, leverages the forward and backward dynamics. Key to our approach is a new analysis which explores the interplay between consistent dynamics and their associated Koopman operators.
  arXiv  Detail & Related papers  (2020-03-04T18:24:30Z)
• How to train your neural ODE: the world of Jacobian and kinetic regularization [7.83405844354125]
  Training neural ODEs on large datasets has not been tractable due to the necessity of allowing the adaptive numerical ODE solver to refine its step size to very small values. We introduce a theoretically-grounded combination of both optimal transport and stability regularizations which encourage neural ODEs to prefer simpler dynamics out of all the dynamics that solve a problem well.
  arXiv  Detail & Related papers  (2020-02-07T14:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.