Enhancing Continuous Time Series Modelling with a Latent ODE-LSTM Approach

• URL: http://arxiv.org/abs/2307.05126v1
• Date: Tue, 11 Jul 2023 09:01:49 GMT
• Title: Enhancing Continuous Time Series Modelling with a Latent ODE-LSTM Approach
• Authors: C. Coelho, M. Fernanda P. Costa, L.L. Ferr\'as
• Abstract summary: Continuous Time Series (CTS) are found in many applications. CTS with irregular sampling rate are difficult to model with standard Recurrent Neural Networks (RNNs)
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Due to their dynamic properties such as irregular sampling rate and high-frequency sampling, Continuous Time Series (CTS) are found in many applications. Since CTS with irregular sampling rate are difficult to model with standard Recurrent Neural Networks (RNNs), RNNs have been generalised to have continuous-time hidden dynamics defined by a Neural Ordinary Differential Equation (Neural ODE), leading to the ODE-RNN model. Another approach that provides a better modelling is that of the Latent ODE model, which constructs a continuous-time model where a latent state is defined at all times. The Latent ODE model uses a standard RNN as the encoder and a Neural ODE as the decoder. However, since the RNN encoder leads to difficulties with missing data and ill-defined latent variables, a Latent ODE-RNN model has recently been proposed that uses a ODE-RNN model as the encoder instead. Both the Latent ODE and Latent ODE-RNN models are difficult to train due to the vanishing and exploding gradients problem. To overcome this problem, the main contribution of this paper is to propose and illustrate a new model based on a new Latent ODE using an ODE-LSTM (Long Short-Term Memory) network as an encoder -- the Latent ODE-LSTM model. To limit the growth of the gradients the Norm Gradient Clipping strategy was embedded on the Latent ODE-LSTM model. The performance evaluation of the new Latent ODE-LSTM (with and without Norm Gradient Clipping) for modelling CTS with regular and irregular sampling rates is then demonstrated. Numerical experiments show that the new Latent ODE-LSTM performs better than Latent ODE-RNNs and can avoid the vanishing and exploding gradients during training.

Related papers

• Path-minimizing Latent ODEs for improved extrapolation and inference [0.0]
  Latent ODE models provide flexible descriptions of dynamic systems, but they can struggle with extrapolation and predicting complicated non-linear dynamics. In this paper we exploit this dichotomy by encouraging time-independent latent representations. By replacing the common variational penalty in latent space with an $ell$ penalty on the path length of each system, the models learn data representations that can easily be distinguished from those of systems with different configurations. This results in faster training, smaller models, more accurate and long-time extrapolation compared to the baseline ODE models with GRU, RNN, and LSTM/decoders on tests with
  arXiv  Detail & Related papers  (2024-10-11T15:50:01Z)
• Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference [60.32804641276217]
  We propose Latent Consistency Models (LCMs), enabling swift inference with minimal steps on any pre-trained LDMs. A high-quality 768 x 768 24-step LCM takes only 32 A100 GPU hours for training. We also introduce Latent Consistency Fine-tuning (LCF), a novel method that is tailored for fine-tuning LCMs on customized image datasets.
  arXiv  Detail & Related papers  (2023-10-06T17:11:58Z)
• Neural Chronos ODE: Unveiling Temporal Patterns and Forecasting Future and Past Trends in Time Series Data [0.0]
  Experimental results demonstrate that Neural CODE outperforms Neural ODE in learning the dynamics of a spiral forward and backward in time. We compare the performance of CODE-RNN/-GRU/-LSTM and CODE-BiRNN/-BiGRU/-BiLSTM against ODE-RNN/-GRU/-LSTM on three real-life time series data tasks.
  arXiv  Detail & Related papers  (2023-07-03T13:54:50Z)
• Deep Latent State Space Models for Time-Series Generation [68.45746489575032]
  We propose LS4, a generative model for sequences with latent variables evolving according to a state space ODE. Inspired by recent deep state space models (S4), we achieve speedups by leveraging a convolutional representation of LS4. We show that LS4 significantly outperforms previous continuous-time generative models in terms of marginal distribution, classification, and prediction scores on real-world datasets.
  arXiv  Detail & Related papers  (2022-12-24T15:17:42Z)
• Improved Batching Strategy For Irregular Time-Series ODE [0.0]
  We propose an improvement in the runtime on ODE-RNNs by using a different efficient strategy. Our experiments show that the new models reduce the runtime of ODE-RNN significantly ranging from 2 times up to 49 times depending on the irregularity of the data.
  arXiv  Detail & Related papers  (2022-07-12T17:30:02Z)
• On the balance between the training time and interpretability of neural ODE for time series modelling [77.34726150561087]
  The paper shows that modern neural ODE cannot be reduced to simpler models for time-series modelling applications. The complexity of neural ODE is compared to or exceeds the conventional time-series modelling tools. We propose a new view on time-series modelling using combined neural networks and an ODE system approach.
  arXiv  Detail & Related papers  (2022-06-07T13:49:40Z)
• Sparse Flows: Pruning Continuous-depth Models [107.98191032466544]
  We show that pruning improves generalization for neural ODEs in generative modeling. We also show that pruning finds minimal and efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy.
  arXiv  Detail & Related papers  (2021-06-24T01:40:17Z)
• Neural ODE Processes [64.10282200111983]
  We introduce Neural ODE Processes (NDPs), a new class of processes determined by a distribution over Neural ODEs. We show that our model can successfully capture the dynamics of low-dimensional systems from just a few data-points.
  arXiv  Detail & Related papers  (2021-03-23T09:32:06Z)
• STEER: Simple Temporal Regularization For Neural ODEs [80.80350769936383]
  We propose a new regularization technique: randomly sampling the end time of the ODE during training. The proposed regularization is simple to implement, has negligible overhead and is effective across a wide variety of tasks. We show through experiments on normalizing flows, time series models and image recognition that the proposed regularization can significantly decrease training time and even improve performance over baseline models.
  arXiv  Detail & Related papers  (2020-06-18T17:44:50Z)
• Learning Long-Term Dependencies in Irregularly-Sampled Time Series [16.762335749650717]
  Recurrent neural networks (RNNs) with continuous-time hidden states are a natural fit for modeling irregularly-sampled time series. We prove that similar to standard RNNs, the underlying reason for this issue is the vanishing or exploding of the gradient during training. We provide a solution by designing a new algorithm based on the long short-term memory (LSTM) that separates its memory from its time-continuous state.
  arXiv  Detail & Related papers  (2020-06-08T08:46:58Z)
• Neural Ordinary Differential Equation based Recurrent Neural Network Model [0.7233897166339269]
  differential equations are a promising new member in the neural network family. This paper explores the strength of the ordinary differential equation (ODE) is explored with a new extension. Two new ODE-based RNN models (GRU-ODE model and LSTM-ODE) can compute the hidden state and cell state at any point of time using an ODE solver. Experiments show that these new ODE based RNN models require less training time than Latent ODEs and conventional Neural ODEs.
  arXiv  Detail & Related papers  (2020-05-20T01:02:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.