An axiomatized PDE model of deep neural networks

• URL: http://arxiv.org/abs/2307.12333v2
• Date: Fri, 22 Mar 2024 14:36:29 GMT
• Title: An axiomatized PDE model of deep neural networks
• Authors: Tangjun Wang, Wenqi Tao, Chenglong Bao, Zuoqiang Shi,
• Abstract summary: Inspired by relation between deep neural network (DNN) and partial differential equations (PDEs), we study the general form of the PDE models of deep neural networks. We prove that the evolution operator is actually determined by convection-diffusion equation. We show that the convection-diffusion equation model improves the robustness and reduces the Rademacher complexity.
• Score: 12.82710074674
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Inspired by the relation between deep neural network (DNN) and partial differential equations (PDEs), we study the general form of the PDE models of deep neural networks. To achieve this goal, we formulate DNN as an evolution operator from a simple base model. Based on several reasonable assumptions, we prove that the evolution operator is actually determined by convection-diffusion equation. This convection-diffusion equation model gives mathematical explanation for several effective networks. Moreover, we show that the convection-diffusion model improves the robustness and reduces the Rademacher complexity. Based on the convection-diffusion equation, we design a new training method for ResNets. Experiments validate the performance of the proposed method.

Related papers

• Automatic Differentiation is Essential in Training Neural Networks for Solving Differential Equations [7.890817997914349]
  Neural network-based approaches have recently shown significant promise in solving partial differential equations (PDEs) in science and engineering. One advantage of the neural network methods for PDEs lies in its automatic differentiation (AD) In this paper, we quantitatively demonstrate the advantage of AD in training neural networks.
  arXiv  Detail & Related papers  (2024-05-23T02:01:05Z)
• Discovering Physics-Informed Neural Networks Model for Solving Partial Differential Equations through Evolutionary Computation [5.8407437499182935]
  This article proposes an evolutionary computation method aimed at discovering the PINNs model with higher approximation accuracy and faster convergence rate. In experiments, the performance of different models that are searched through Bayesian optimization, random search and evolution is compared in solving Klein-Gordon, Burgers, and Lam'e equations.
  arXiv  Detail & Related papers  (2024-05-18T07:32:02Z)
• Diffusion models as probabilistic neural operators for recovering unobserved states of dynamical systems [49.2319247825857]
  We show that diffusion-based generative models exhibit many properties favourable for neural operators. We propose to train a single model adaptable to multiple tasks, by alternating between the tasks during training.
  arXiv  Detail & Related papers  (2024-05-11T21:23:55Z)
• Convection-Diffusion Equation: A Theoretically Certified Framework for Neural Networks [14.01268607317875]
  We study the partial differential equation models of neural networks. We show that this map can be formulated by a convection-diffusion equation. We design a novel network structure, which incorporates diffusion mechanism into network architecture.
  arXiv  Detail & Related papers  (2024-03-23T05:26:36Z)
• Neural Basis Functions for Accelerating Solutions to High Mach Euler Equations [63.8376359764052]
  We propose an approach to solving partial differential equations (PDEs) using a set of neural networks. We regress a set of neural networks onto a reduced order Proper Orthogonal Decomposition (POD) basis. These networks are then used in combination with a branch network that ingests the parameters of the prescribed PDE to compute a reduced order approximation to the PDE.
  arXiv  Detail & Related papers  (2022-08-02T18:27:13Z)
• Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEs [70.51212431290611]
  Partial differential equations (SPDEs) are significant tools for modeling dynamics in many areas including atmospheric sciences and physics. We propose the Neural Operator with Regularity Structure (NORS) which incorporates the feature vectors for modeling dynamics driven by SPDEs. We conduct experiments on various of SPDEs including the dynamic Phi41 model and the 2d Navier-Stokes equation.
  arXiv  Detail & Related papers  (2022-04-13T08:53:41Z)
• Learning Physics-Informed Neural Networks without Stacked Back-propagation [82.26566759276105]
  We develop a novel approach that can significantly accelerate the training of Physics-Informed Neural Networks. In particular, we parameterize the PDE solution by the Gaussian smoothed model and show that, derived from Stein's Identity, the second-order derivatives can be efficiently calculated without back-propagation. Experimental results show that our proposed method can achieve competitive error compared to standard PINN training but is two orders of magnitude faster.
  arXiv  Detail & Related papers  (2022-02-18T18:07:54Z)
• Climate Modeling with Neural Diffusion Equations [3.8521112392276]
  We design a novel climate model based on the neural ordinary differential equation (NODE) and the diffusion equation. Our method consistently outperforms existing baselines by non-trivial margins.
  arXiv  Detail & Related papers  (2021-11-11T01:48:46Z)
• SPINN: Sparse, Physics-based, and Interpretable Neural Networks for PDEs [0.0]
  We introduce a class of Sparse, Physics-based, and Interpretable Neural Networks (SPINN) for solving ordinary and partial differential equations. By reinterpreting a traditional meshless representation of solutions of PDEs as a special sparse deep neural network, we develop a class of sparse neural network architectures that are interpretable.
  arXiv  Detail & Related papers  (2021-02-25T17:45:50Z)
• Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction [79.81193813215872]
  We develop a hybrid (graph) neural network that combines a traditional graph convolutional network with an embedded differentiable fluid dynamics simulator inside the network itself. We show that we can both generalize well to new situations and benefit from the substantial speedup of neural network CFD predictions.
  arXiv  Detail & Related papers  (2020-07-08T21:23:19Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.