Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations

• URL: http://arxiv.org/abs/2101.03583v1
• Date: Sun, 10 Jan 2021 17:34:10 GMT
• Title: Accuracy and Architecture Studies of Residual Neural Network solving Ordinary Differential Equations
• Authors: Changxin Qiu, Aaron Bendickson, Joshua Kalyanapu and Jue Yan
• Abstract summary: We consider utilizing a residual neural network (ResNet) to solve ordinary differential equations. We apply forward Euler, Runge-Kutta2 and Runge-Kutta4 finite difference methods to generate three sets of targets training the ResNet. The well trained ResNet behaves just as its counterpart of the corresponding one-step finite difference method.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper we consider utilizing a residual neural network (ResNet) to solve ordinary differential equations. Stochastic gradient descent method is applied to obtain the optimal parameter set of weights and biases of the network. We apply forward Euler, Runge-Kutta2 and Runge-Kutta4 finite difference methods to generate three sets of targets training the ResNet and carry out the target study. The well trained ResNet behaves just as its counterpart of the corresponding one-step finite difference method. In particular, we carry out (1) the architecture study in terms of number of hidden layers and neurons per layer to find the optimal ResNet structure; (2) the target study to verify the ResNet solver behaves as accurate as its finite difference method counterpart; (3) solution trajectory simulation. Even the ResNet solver looks like and is implemented in a way similar to forward Euler scheme, its accuracy can be as high as any one step method. A sequence of numerical examples are presented to demonstrate the performance of the ResNet solver.

Related papers

• Rational-WENO: A lightweight, physically-consistent three-point weighted essentially non-oscillatory scheme [14.120671138290104]
  We employ a rational neural network to accurately estimate the local smoothness of the solution. This approach achieves a granular reconstruction with significantly reduced dissipation. We demonstrate the effectiveness of our approach on several one-, two-, and three-dimensional fluid flow problems.
  arXiv  Detail & Related papers  (2024-09-13T22:11:03Z)
• Error Analysis of Three-Layer Neural Network Trained with PGD for Deep Ritz Method [7.723218675113336]
  We employ a three-layer tanh neural network within the framework of the deep Ritz method to solve second-order elliptic equations. We perform projected gradient descent to train the three-layer network and we establish its global convergence. We present error bound in terms of the sample size $n$ and our work provides guidance on how to set the network depth, width, step size, and number of iterations for the projected gradient descent algorithm.
  arXiv  Detail & Related papers  (2024-05-19T05:07:09Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Message Passing Neural PDE Solvers [60.77761603258397]
  We build a neural message passing solver, replacing allally designed components in the graph with backprop-optimized neural function approximators. We show that neural message passing solvers representationally contain some classical methods, such as finite differences, finite volumes, and WENO schemes. We validate our method on various fluid-like flow problems, demonstrating fast, stable, and accurate performance across different domain topologies, equation parameters, discretizations, etc., in 1D and 2D.
  arXiv  Detail & Related papers  (2022-02-07T17:47:46Z)
• Joint inference and input optimization in equilibrium networks [68.63726855991052]
  deep equilibrium model is a class of models that foregoes traditional network depth and instead computes the output of a network by finding the fixed point of a single nonlinear layer. We show that there is a natural synergy between these two settings. We demonstrate this strategy on various tasks such as training generative models while optimizing over latent codes, training models for inverse problems like denoising and inpainting, adversarial training and gradient based meta-learning.
  arXiv  Detail & Related papers  (2021-11-25T19:59:33Z)
• Meta-Solver for Neural Ordinary Differential Equations [77.8918415523446]
  We investigate how the variability in solvers' space can improve neural ODEs performance. We show that the right choice of solver parameterization can significantly affect neural ODEs models in terms of robustness to adversarial attacks.
  arXiv  Detail & Related papers  (2021-03-15T17:26:34Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Learning Neural Network Subspaces [74.44457651546728]
  Recent observations have advanced our understanding of the neural network optimization landscape. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks.
  arXiv  Detail & Related papers  (2021-02-20T23:26:58Z)
• Computational characteristics of feedforward neural networks for solving a stiff differential equation [0.0]
  We study the solution of a simple but fundamental stiff ordinary differential equation modelling a damped system. We show that it is possible to identify preferable choices to be made for parameters and methods. Overall we extend the current literature in the field by showing what can be done in order to obtain reliable and accurate results by the neural network approach.
  arXiv  Detail & Related papers  (2020-12-03T12:22:24Z)
• A Residual Solver and Its Unfolding Neural Network for Total Variation Regularized Models [5.9622541907827875]
  This paper proposes to solve the Total Variation regularized models by finding the residual between the input and the unknown optimal solution. We numerically confirm that the residual solver can reach the same global optimal solutions as the classical method on 500 natural images. Both the proposed algorithm and neural network are successfully applied on several problems to demonstrate their effectiveness and efficiency.
  arXiv  Detail & Related papers  (2020-09-08T01:44:34Z)
• ODEN: A Framework to Solve Ordinary Differential Equations using Artificial Neural Networks [0.0]
  We prove a specific loss function, which does not require knowledge of the exact solution, to evaluate neural networks' performance. Neural networks are shown to be proficient at approximating continuous solutions within their training domains. A user-friendly and adaptable open-source code (ODE$mathcalN$) is provided on GitHub.
  arXiv  Detail & Related papers  (2020-05-28T15:34:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.