Stability of implicit neural networks for long-term forecasting in dynamical systems

• URL: http://arxiv.org/abs/2305.17155v2
• Date: Thu, 8 Jun 2023 13:29:44 GMT
• Title: Stability of implicit neural networks for long-term forecasting in dynamical systems
• Authors: Leon Migus, Julien Salomon and Patrick Gallinari
• Abstract summary: We develop a theory based on the stability definition of schemes to ensure the stability in forecasting of this network. Our experimental results validate our stability property, and show improved results at long-term forecasting for two transports PDEs.
• Score: 10.74610263406029
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Forecasting physical signals in long time range is among the most challenging tasks in Partial Differential Equations (PDEs) research. To circumvent limitations of traditional solvers, many different Deep Learning methods have been proposed. They are all based on auto-regressive methods and exhibit stability issues. Drawing inspiration from the stability property of implicit numerical schemes, we introduce a stable auto-regressive implicit neural network. We develop a theory based on the stability definition of schemes to ensure the stability in forecasting of this network. It leads us to introduce hard constraints on its weights and propagate the dynamics in the latent space. Our experimental results validate our stability property, and show improved results at long-term forecasting for two transports PDEs.

Related papers

• Some Results on Neural Network Stability, Consistency, and Convergence: Insights into Non-IID Data, High-Dimensional Settings, and Physics-Informed Neural Networks [0.0]
  This paper addresses critical challenges in machine learning, particularly under non-IID data distribution conditions. We provide results on uniform stability networks with dynamic learning rates. These results fill significant gaps in understanding model behavior in noisy environments.
  arXiv  Detail & Related papers  (2024-09-08T08:48:42Z)
• Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data [3.686808512438363]
  Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods. We propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.
  arXiv  Detail & Related papers  (2024-02-22T22:00:03Z)
• Uncertainty Quantification for Forward and Inverse Problems of PDEs via Latent Global Evolution [110.99891169486366]
  We propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model. Our method endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems. Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions.
  arXiv  Detail & Related papers  (2024-02-13T11:22:59Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks [12.355137704908042]
  We show restrained numerical instabilities in current training practices of deep networks with gradient descent (SGD) We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of convolutional neural networks (CNNs) We show this is a consequence of the non-linear PDE associated with the descent of the CNN, whose local linearization changes when over-driving the step size of the discretization resulting in a stabilizing effect.
  arXiv  Detail & Related papers  (2022-06-04T14:54:05Z)
• Accurate and Reliable Forecasting using Stochastic Differential Equations [48.21369419647511]
  It is critical yet challenging for deep learning models to properly characterize uncertainty that is pervasive in real-world environments. This paper develops SDE-HNN to characterize the interaction between the predictive mean and variance of HNNs for accurate and reliable regression. Experiments on the challenging datasets show that our method significantly outperforms the state-of-the-art baselines in terms of both predictive performance and uncertainty quantification.
  arXiv  Detail & Related papers  (2021-03-28T04:18:11Z)
• Almost Surely Stable Deep Dynamics [4.199844472131922]
  We introduce a method for learning provably stable deep neural network based dynamic models from observed data. Our method works by embedding a Lyapunov neural network into the dynamic model, thereby inherently satisfying the stability criterion.
  arXiv  Detail & Related papers  (2021-03-26T20:37:08Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.