Divide And Conquer: Learning Chaotic Dynamical Systems With Multistep Penalty Neural Ordinary Differential Equations

• URL: http://arxiv.org/abs/2407.00568v2
• Date: Tue, 2 Jul 2024 02:54:49 GMT
• Title: Divide And Conquer: Learning Chaotic Dynamical Systems With Multistep Penalty Neural Ordinary Differential Equations
• Authors: Dibyajyoti Chakraborty, Seung Whan Chung, Romit Maulik,
• Abstract summary: Multistep Penalty NODE(MP-NODE) is applied to chaotic systems such as the Kuramoto-Sivashinsky equation and the two-dimensional Kolmogorov flow. It can optimize chaotic systems in a manner similar to least-squares landscape with significantly lower computational costs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Forecasting high-dimensional dynamical systems is a fundamental challenge in various fields, such as the geosciences and engineering. Neural Ordinary Differential Equations (NODEs), which combine the power of neural networks and numerical solvers, have emerged as a promising algorithm for forecasting complex nonlinear dynamical systems. However, classical techniques used for NODE training are ineffective for learning chaotic dynamical systems. In this work, we propose a novel NODE-training approach that allows for robust learning of chaotic dynamical systems. Our method addresses the challenges of non-convexity and exploding gradients associated with underlying chaotic dynamics. Training data trajectories from such systems are split into multiple, non-overlapping time windows. In addition to the deviation from the training data, the optimization loss term further penalizes the discontinuities of the predicted trajectory between the time windows. The window size is selected based on the fastest Lyapunov time scale of the system. Multi-step penalty(MP) method is first demonstrated on Lorenz equation, to illustrate how it improves the loss landscape and thereby accelerating the optimization convergence. MP method can optimize chaotic systems in a manner similar to least-squares shadowing with significantly lower computational costs. Our proposed algorithm, denoted the Multistep Penalty NODE(MP-NODE), is applied to chaotic systems such as the Kuramoto-Sivashinsky equation and the two-dimensional Kolmogorov flow. It is observed that MP-NODE provide viable performance for such chaotic systems, not only for short-term trajectory predictions but also for invariant statistics that are hallmarks of the chaotic nature of these dynamics.

Related papers

• Latent Neural PDE Solver: a reduced-order modelling framework for partial differential equations [6.173339150997772]
  We propose to learn the dynamics of the system in the latent space with much coarser discretizations. A non-linear autoencoder is first trained to project the full-order representation of the system onto the mesh-reduced space. We showcase that it has competitive accuracy and efficiency compared to the neural PDE solver that operates on full-order space.
  arXiv  Detail & Related papers  (2024-02-27T19:36:27Z)
• Constraining Chaos: Enforcing dynamical invariants in the training of recurrent neural networks [0.0]
  We introduce a novel training method for machine learning based forecasting methods for chaotic dynamical systems. The training enforces dynamical invariants--such as the Lyapunov exponent spectrum and fractal dimension--in the systems of interest, enabling longer and more stable forecasts when operating with limited data.
  arXiv  Detail & Related papers  (2023-04-24T00:33:47Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Losing momentum in continuous-time stochastic optimisation [62.997667081978825]
  momentum-based algorithms have become especially popular in recent years. In work, we propose and analyse a continuous-time model for gradient descent with momentum. We show convergence of our system to the global minimiser when reducing momentum over time.
  arXiv  Detail & Related papers  (2022-09-08T10:46:05Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• Continual Learning of Dynamical Systems with Competitive Federated Reservoir Computing [29.98127520773633]
  Continual learning aims to rapidly adapt to abrupt system changes without previous dynamical regimes. This work proposes an approach to continual learning based reservoir computing. We show that this multi-head reservoir minimizes interference and forgetting on several dynamical systems.
  arXiv  Detail & Related papers  (2022-06-27T14:35:50Z)
• Learning effective dynamics from data-driven stochastic systems [2.4578723416255754]
  This work is devoted to investigating the effective dynamics for slow-fast dynamical systems. We propose a novel algorithm including a neural network called Auto-SDE to learn in slow manifold.
  arXiv  Detail & Related papers  (2022-05-09T09:56:58Z)
• DiffPD: Differentiable Projective Dynamics with Contact [65.88720481593118]
  We present DiffPD, an efficient differentiable soft-body simulator with implicit time integration. We evaluate the performance of DiffPD and observe a speedup of 4-19 times compared to the standard Newton's method in various applications.
  arXiv  Detail & Related papers  (2021-01-15T00:13:33Z)
• Hierarchical Deep Learning of Multiscale Differential Equation Time-Steppers [5.6385744392820465]
  We develop a hierarchy of deep neural network time-steppers to approximate the flow map of the dynamical system over a disparate range of time-scales. The resulting model is purely data-driven and leverages features of the multiscale dynamics. We benchmark our algorithm against state-of-the-art methods, such as LSTM, reservoir computing, and clockwork RNN.
  arXiv  Detail & Related papers  (2020-08-22T07:16:53Z)
• Reinforcement Learning with Fast Stabilization in Linear Dynamical Systems [91.43582419264763]
  We study model-based reinforcement learning (RL) in unknown stabilizable linear dynamical systems. We propose an algorithm that certifies fast stabilization of the underlying system by effectively exploring the environment. We show that the proposed algorithm attains $tildemathcalO(sqrtT)$ regret after $T$ time steps of agent-environment interaction.
  arXiv  Detail & Related papers  (2020-07-23T23:06:40Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.