Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization

• URL: http://arxiv.org/abs/2211.05262v1
• Date: Wed, 9 Nov 2022 23:40:52 GMT
• Title: Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization
• Authors: Alexander Wikner, Brian R. Hunt, Joseph Harvey, Michelle Girvan, Edward Ott
• Abstract summary: Recent has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of chaotic dynamical systems. In the absence of mitigating techniques, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. We introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training.
• Score: 58.720142291102135
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of unknown chaotic dynamical systems. Such ML models can be used to produce both short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics (``climate''). Both of these tasks can be accomplished by employing a feedback loop, whereby the model is trained to predict forward one time step, then the trained model is iterated for multiple time steps with its output used as the input. In the absence of mitigating techniques, however, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. In this article, we systematically examine the technique of adding noise to the ML model input during training as a means to promote stability and improve prediction accuracy. Furthermore, we introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. Our case study uses reservoir computing, a machine-learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto-Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while reservoir computers trained without regularization are unstable. Compared with other types of regularization that yield stability in some cases, we find that both short-term and climate predictions from reservoir computers trained with noise or with LMNT are substantially more accurate. Finally, we show that the deterministic aspect of our LMNT regularization facilitates fast hyperparameter tuning when compared to training with noise.

Related papers

• Ensemble models outperform single model uncertainties and predictions for operator-learning of hypersonic flows [43.148818844265236]
  Training scientific machine learning (SciML) models on limited high-fidelity data offers one approach to rapidly predict behaviors for situations that have not been seen before. High-fidelity data is itself in limited quantity to validate all outputs of the SciML model in unexplored input space. We extend a DeepONet using three different uncertainty mechanisms: mean-variance estimation, evidential uncertainty, and ensembling.
  arXiv  Detail & Related papers  (2023-10-31T18:07:29Z)
• Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling [13.38194491846739]
  We propose a novel machine learning model based on Koopman operator theory, which we call Koopman Invertible Autoencoders (KIA) KIA captures the inherent characteristic of the system by modeling both forward and backward dynamics in the infinite-dimensional Hilbert space. This enables us to efficiently learn low-dimensional representations, resulting in more accurate predictions of long-term system behavior.
  arXiv  Detail & Related papers  (2023-09-19T03:42:55Z)
• Physics-Inspired Temporal Learning of Quadrotor Dynamics for Accurate Model Predictive Trajectory Tracking [76.27433308688592]
  Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation. We present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience. Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions.
  arXiv  Detail & Related papers  (2022-06-07T13:51:35Z)
• Learning continuous models for continuous physics [94.42705784823997]
  We develop a test based on numerical analysis theory to validate machine learning models for science and engineering applications. Our results illustrate how principled numerical analysis methods can be coupled with existing ML training/testing methodologies to validate models for science and engineering applications.
  arXiv  Detail & Related papers  (2022-02-17T07:56:46Z)
• Using scientific machine learning for experimental bifurcation analysis of dynamic systems [2.204918347869259]
  This study focuses on training universal differential equation (UDE) models for physical nonlinear dynamical systems with limit cycles. We consider examples where training data is generated by numerical simulations, whereas we also employ the proposed modelling concept to physical experiments. We use both neural networks and Gaussian processes as universal approximators alongside the mechanistic models to give a critical assessment of the accuracy and robustness of the UDE modelling approach.
  arXiv  Detail & Related papers  (2021-10-22T15:43:03Z)
• Data-driven method for real-time prediction and uncertainty quantification of fatigue failure under stochastic loading using artificial neural networks and Gaussian process regression [0.0]
  Methods for early failure prediction are essential for engineering, military, and civil applications. Uncertainty (UQ) is of major importance for real-time decision-making purposes.
  arXiv  Detail & Related papers  (2021-03-11T13:57:08Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• Short- and long-term prediction of a chaotic flow: A physics-constrained reservoir computing approach [5.37133760455631]
  We propose a physics-constrained machine learning method-based on reservoir computing- to time-accurately predict extreme events and long-term velocity statistics in a model of turbulent shear flow. We show that the combination of the two approaches is able to accurately reproduce the velocity statistics and to predict the occurrence and amplitude of extreme events in a model of self-sustaining process in turbulence.
  arXiv  Detail & Related papers  (2021-02-15T12:29:09Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.