Contracting Neural-Newton Solver

• URL: http://arxiv.org/abs/2106.02543v1
• Date: Fri, 4 Jun 2021 15:14:12 GMT
• Title: Contracting Neural-Newton Solver
• Authors: Samuel Chevalier, Jochen Stiasny, Spyros Chatzivasileiadis
• Abstract summary: We develop a recurrent NN simulation tool, termed the Contracting Neural-Newton Solver (CoNNS) In this paper, we model the Newton solver at the heart of an implicit Runge-Kutta integrator as a contracting map iteratively seeking this fixed point. We prove that successive passes through the NN are guaranteed to converge to a unique, fixed point.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in deep learning have set the focus on neural networks (NNs) that can successfully replace traditional numerical solvers in many applications, achieving impressive computing gains. One such application is time domain simulation, which is indispensable for the design, analysis and operation of many engineering systems. Simulating dynamical systems with implicit Newton-based solvers is a computationally heavy task, as it requires the solution of a parameterized system of differential and algebraic equations at each time step. A variety of NN-based methodologies have been shown to successfully approximate the dynamical trajectories computed by numerical time domain solvers at a fraction of the time. However, so far no previous NN-based model has explicitly captured the fact that any predicted point on the time domain trajectory also represents the fixed point of the numerical solver itself. As we show, explicitly capturing this property can lead to significantly increased NN accuracy and much smaller NN sizes. In this paper, we model the Newton solver at the heart of an implicit Runge-Kutta integrator as a contracting map iteratively seeking this fixed point. Our primary contribution is to develop a recurrent NN simulation tool, termed the Contracting Neural-Newton Solver (CoNNS), which explicitly captures the contracting nature of these Newton iterations. To build CoNNS, we train a feedforward NN and mimic this contraction behavior by embedding a series of training constraints which guarantee the mapping provided by the NN satisfies the Banach fixed-point theorem; thus, we are able to prove that successive passes through the NN are guaranteed to converge to a unique, fixed point.

Related papers

• Scalable Mechanistic Neural Networks [52.28945097811129]
  We propose an enhanced neural network framework designed for scientific machine learning applications involving long temporal sequences. By reformulating the original Mechanistic Neural Network (MNN) we reduce the computational time and space complexities from cubic and quadratic with respect to the sequence length, respectively, to linear. Extensive experiments demonstrate that S-MNN matches the original MNN in precision while substantially reducing computational resources.
  arXiv  Detail & Related papers  (2024-10-08T14:27:28Z)
• Correctness Verification of Neural Networks Approximating Differential Equations [0.0]
  Neural Networks (NNs) approximate the solution of Partial Differential Equations (PDEs) NNs can become integral parts of simulation software tools which can accelerate the simulation of complex dynamic systems more than 100 times. This work addresses the verification of these functions by defining the NN derivative as a finite difference approximation. For the first time, we tackle the problem of bounding an NN function without a priori knowledge of the output domain.
  arXiv  Detail & Related papers  (2024-02-12T12:55:35Z)
• Analyzing Convergence in Quantum Neural Networks: Deviations from Neural Tangent Kernels [20.53302002578558]
  A quantum neural network (QNN) is a parameterized mapping efficiently implementable on near-term Noisy Intermediate-Scale Quantum (NISQ) computers. Despite the existing empirical and theoretical investigations, the convergence of QNN training is not fully understood.
  arXiv  Detail & Related papers  (2023-03-26T22:58:06Z)
• Learning Trajectories of Hamiltonian Systems with Neural Networks [81.38804205212425]
  We propose to enhance Hamiltonian neural networks with an estimation of a continuous-time trajectory of the modeled system. We demonstrate that the proposed integration scheme works well for HNNs, especially with low sampling rates, noisy and irregular observations.
  arXiv  Detail & Related papers  (2022-04-11T13:25:45Z)
• A unified framework for Hamiltonian deep neural networks [3.0934684265555052]
  Training deep neural networks (DNNs) can be difficult due to vanishing/exploding gradients during weight optimization. We propose a class of DNNs stemming from the time discretization of Hamiltonian systems. The proposed Hamiltonian framework, besides encompassing existing networks inspired by marginally stable ODEs, allows one to derive new and more expressive architectures.
  arXiv  Detail & Related papers  (2021-04-27T13:20:24Z)
• dNNsolve: an efficient NN-based PDE solver [62.997667081978825]
  We introduce dNNsolve, that makes use of dual Neural Networks to solve ODEs/PDEs. We show that dNNsolve is capable of solving a broad range of ODEs/PDEs in 1, 2 and 3 spacetime dimensions.
  arXiv  Detail & Related papers  (2021-03-15T19:14:41Z)
• UnICORNN: A recurrent model for learning very long time dependencies [0.0]
  We propose a novel RNN architecture based on a structure preserving discretization of a Hamiltonian system of second-order ordinary differential equations. The resulting RNN is fast, invertible (in time), memory efficient and we derive rigorous bounds on the hidden state gradients to prove the mitigation of the exploding and vanishing gradient problem.
  arXiv  Detail & Related papers  (2021-03-09T15:19:59Z)
• Toward Trainability of Quantum Neural Networks [87.04438831673063]
  Quantum Neural Networks (QNNs) have been proposed as generalizations of classical neural networks to achieve the quantum speed-up. Serious bottlenecks exist for training QNNs due to the vanishing with gradient rate exponential to the input qubit number. We show that QNNs with tree tensor and step controlled structures for the application of binary classification. Simulations show faster convergent rates and better accuracy compared to QNNs with random structures.
  arXiv  Detail & Related papers  (2020-11-12T08:32:04Z)
• Absence of Barren Plateaus in Quantum Convolutional Neural Networks [0.0]
  Quantum Convolutional Neural Networks (QCNNs) have been proposed. We rigorously analyze the gradient scaling for the parameters in the QCNN architecture.
  arXiv  Detail & Related papers  (2020-11-05T16:46:13Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.