Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm

• URL: http://arxiv.org/abs/2207.12242v1
• Date: Thu, 7 Jul 2022 23:14:12 GMT
• Title: Multisymplectic Formulation of Deep Learning Using Mean--Field Type Control and Nonlinear Stability of Training Algorithm
• Authors: Nader Ganaba
• Abstract summary: We formulate training of deep neural networks as a hydrodynamics system with a multisymplectic structure. For that, the deep neural network is modelled using a differential equation and, thereby, mean-field type control is used to train it. The numerical scheme, yields an approximated solution which is also an exact solution of a hydrodynamics system with a multisymplectic structure.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As it stands, a robust mathematical framework to analyse and study various topics in deep learning is yet to come to the fore. Nonetheless, viewing deep learning as a dynamical system allows the use of established theories to investigate the behaviour of deep neural networks. In order to study the stability of the training process, in this article, we formulate training of deep neural networks as a hydrodynamics system, which has a multisymplectic structure. For that, the deep neural network is modelled using a stochastic differential equation and, thereby, mean-field type control is used to train it. The necessary conditions for optimality of the mean--field type control reduce to a system of Euler-Poincare equations, which has the a similar geometric structure to that of compressible fluids. The mean-field type control is solved numerically using a multisymplectic numerical scheme that takes advantage of the underlying geometry. Moreover, the numerical scheme, yields an approximated solution which is also an exact solution of a hydrodynamics system with a multisymplectic structure and it can be analysed using backward error analysis. Further, nonlinear stability yields the condition for selecting the number of hidden layers and the number of nodes per layer, that makes the training stable while approximating the solution of a residual neural network with a number of hidden layers approaching infinity.

Related papers

• Learning Controlled Stochastic Differential Equations [61.82896036131116]
  This work proposes a novel method for estimating both drift and diffusion coefficients of continuous, multidimensional, nonlinear controlled differential equations with non-uniform diffusion. We provide strong theoretical guarantees, including finite-sample bounds for (L2), (Linfty), and risk metrics, with learning rates adaptive to coefficients' regularity. Our method is available as an open-source Python library.
  arXiv  Detail & Related papers  (2024-11-04T11:09:58Z)
• Deep multitask neural networks for solving some stochastic optimal control problems [0.0]
  In this paper, we consider a class of optimal control problems and introduce an effective solution employing neural networks. To train our multitask neural network, we introduce a novel scheme that dynamically balances the learning across tasks. Through numerical experiments on real-world derivatives pricing problems, we prove that our method outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-01-23T17:20:48Z)
• Solving the Discretised Multiphase Flow Equations with Interface Capturing on Structured Grids Using Machine Learning Libraries [0.6299766708197884]
  This paper solves the discretised multiphase flow equations using tools and methods from machine-learning libraries. For the first time, finite element discretisations of multiphase flows can be solved using an approach based on (untrained) convolutional neural networks.
  arXiv  Detail & Related papers  (2024-01-12T18:42:42Z)
• Hierarchical deep learning-based adaptive time-stepping scheme for multiscale simulations [0.0]
  This study proposes a new method for simulating multiscale problems using deep neural networks. By leveraging the hierarchical learning of neural network time steppers, the method adapts time steps to approximate dynamical system flow maps across timescales. This approach achieves state-of-the-art performance in less computational time compared to fixed-step neural network solvers.
  arXiv  Detail & Related papers  (2023-11-10T09:47:58Z)
• Implicit regularization of deep residual networks towards neural ODEs [8.075122862553359]
  We establish an implicit regularization of deep residual networks towards neural ODEs. We prove that if the network is as a discretization of a neural ODE, then such a discretization holds throughout training.
  arXiv  Detail & Related papers  (2023-09-03T16:35:59Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Message Passing Neural PDE Solvers [60.77761603258397]
  We build a neural message passing solver, replacing allally designed components in the graph with backprop-optimized neural function approximators. We show that neural message passing solvers representationally contain some classical methods, such as finite differences, finite volumes, and WENO schemes. We validate our method on various fluid-like flow problems, demonstrating fast, stable, and accurate performance across different domain topologies, equation parameters, discretizations, etc., in 1D and 2D.
  arXiv  Detail & Related papers  (2022-02-07T17:47:46Z)
• Deep-learning-based upscaling method for geologic models via theory-guided convolutional neural network [0.0]
  A deep convolutional neural network (CNN) is trained to approximate the relationship between the coarse grid of hydraulic conductivity fields and the hydraulic heads. With the physical information considered, dependence on the data volume of training the deep CNN model can be reduced greatly.
  arXiv  Detail & Related papers  (2021-12-31T08:10:48Z)
• Large-scale Neural Solvers for Partial Differential Equations [48.7576911714538]
  Solving partial differential equations (PDE) is an indispensable part of many branches of science as many processes can be modelled in terms of PDEs. Recent numerical solvers require manual discretization of the underlying equation as well as sophisticated, tailored code for distributed computing. We examine the applicability of continuous, mesh-free neural solvers for partial differential equations, physics-informed neural networks (PINNs) We discuss the accuracy of GatedPINN with respect to analytical solutions -- as well as state-of-the-art numerical solvers, such as spectral solvers.
  arXiv  Detail & Related papers  (2020-09-08T13:26:51Z)
• An Ode to an ODE [78.97367880223254]
  We present a new paradigm for Neural ODE algorithms, called ODEtoODE, where time-dependent parameters of the main flow evolve according to a matrix flow on the group O(d) This nested system of two flows provides stability and effectiveness of training and provably solves the gradient vanishing-explosion problem.
  arXiv  Detail & Related papers  (2020-06-19T22:05:19Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.