Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders

• URL: http://arxiv.org/abs/2301.02790v1
• Date: Sat, 7 Jan 2023 06:31:28 GMT
• Title: Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders
• Authors: Mayank Deshpande, Siddharth Agarwal, Vukka Snigdha, Arya Kumar Bhattacharya
• Abstract summary: An important inference from Neural Kernel Tangent (NTK) theory is the existence of spectral bias (SB) SB is low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. It is firmly established that under normalized conditions, PINNs do exhibit strong spectral bias, and this increases with the order of the differential equation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An important inference from Neural Tangent Kernel (NTK) theory is the existence of spectral bias (SB), that is, low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. Physics Informed Neural Networks (PINNs) are designed to learn the solutions of differential equations (DE) of arbitrary orders; in PINNs the loss functions are obtained as the residues of the conservative form of the DEs and represent the degree of dissatisfaction of the equations. So there has been an open question whether (a) PINNs also exhibit SB and (b) if so, how does this bias vary across the orders of the DEs. In this work, a series of numerical experiments are conducted on simple sinusoidal functions of varying frequencies, compositions and equation orders to investigate these issues. It is firmly established that under normalized conditions, PINNs do exhibit strong spectral bias, and this increases with the order of the differential equation.

Related papers

• Understanding the dynamics of the frequency bias in neural networks [0.0]
  Recent works have shown that traditional Neural Network (NN) architectures display a marked frequency bias in the learning process. We develop a partial differential equation (PDE) that unravels the frequency dynamics of the error for a 2-layer NN. We empirically show that the same principle extends to multi-layer NNs.
  arXiv  Detail & Related papers  (2024-05-23T18:09:16Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Learning Discretized Neural Networks under Ricci Flow [51.36292559262042]
  We study Discretized Neural Networks (DNNs) composed of low-precision weights and activations. DNNs suffer from either infinite or zero gradients due to the non-differentiable discrete function during training.
  arXiv  Detail & Related papers  (2023-02-07T10:51:53Z)
• Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs [86.35471039808023]
  We introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases the number of frequency modes used by the model. We show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.
  arXiv  Detail & Related papers  (2022-11-28T09:57:15Z)
• Neural tangent kernel analysis of PINN for advection-diffusion equation [0.0]
  Physics-informed neural networks (PINNs) numerically approximate the solution of a partial differential equation (PDE) PINNs are known to struggle even in simple cases where the closed-form analytical solution is available. This work focuses on a systematic analysis of PINNs for the linear advection-diffusion equation (LAD) using the Neural Tangent Kernel (NTK) theory.
  arXiv  Detail & Related papers  (2022-11-21T18:35:14Z)
• Tunable Complexity Benchmarks for Evaluating Physics-Informed Neural Networks on Coupled Ordinary Differential Equations [64.78260098263489]
  In this work, we assess the ability of physics-informed neural networks (PINNs) to solve increasingly-complex coupled ordinary differential equations (ODEs) We show that PINNs eventually fail to produce correct solutions to these benchmarks as their complexity increases. We identify several reasons why this may be the case, including insufficient network capacity, poor conditioning of the ODEs, and high local curvature, as measured by the Laplacian of the PINN loss.
  arXiv  Detail & Related papers  (2022-10-14T15:01:32Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• The Spectral Bias of Polynomial Neural Networks [63.27903166253743]
  Polynomial neural networks (PNNs) have been shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a $textitspectral bias$ towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Tangent Kernel (NTK) of PNNs. We find that the $Pi$-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the
  arXiv  Detail & Related papers  (2022-02-27T23:12:43Z)
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks [0.0]
  We show that neural networks (PINNs) struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. We construct novel architectures that employ multi-scale random observational features and justify how such coordinate embedding layers can lead to robust and accurate PINN models.
  arXiv  Detail & Related papers  (2020-12-18T04:19:30Z)
• When and why PINNs fail to train: A neural tangent kernel perspective [2.1485350418225244]
  We derive the Neural Tangent Kernel (NTK) of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. We find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. We propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error.
  arXiv  Detail & Related papers  (2020-07-28T23:44:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.