Convergence analysis of wide shallow neural operators within the framework of Neural Tangent Kernel

• URL: http://arxiv.org/abs/2412.05545v3
• Date: Fri, 10 Jan 2025 14:51:06 GMT
• Title: Convergence analysis of wide shallow neural operators within the framework of Neural Tangent Kernel
• Authors: Xianliang Xu, Ye Li, Zhongyi Huang,
• Abstract summary: We conduct the convergence analysis of gradient descent for the wide shallow neural operators and physics-informed shallow neural operators within the framework of Neural Tangent Kernel (NTK) Under the setting of over-parametrization, gradient descent can find the global minimum regardless of whether it is in continuous time or discrete time.
• Score: 4.313136216120379
• License:
• Abstract: Neural operators are aiming at approximating operators mapping between Banach spaces of functions, achieving much success in the field of scientific computing. Compared to certain deep learning-based solvers, such as Physics-Informed Neural Networks (PINNs), Deep Ritz Method (DRM), neural operators can solve a class of Partial Differential Equations (PDEs). Although much work has been done to analyze the approximation and generalization error of neural operators, there is still a lack of analysis on their training error. In this work, we conduct the convergence analysis of gradient descent for the wide shallow neural operators and physics-informed shallow neural operators within the framework of Neural Tangent Kernel (NTK). The core idea lies on the fact that over-parameterization and random initialization together ensure that each weight vector remains near its initialization throughout all iterations, yielding the linear convergence of gradient descent. In this work, we demonstrate that under the setting of over-parametrization, gradient descent can find the global minimum regardless of whether it is in continuous time or discrete time.

Related papers

• Optimal Convergence Rates for Neural Operators [2.9388890036358104]
  We provide bounds on the number of hidden neurons and the number of second-stage samples necessary for generalization. A key application of neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2024-12-23T12:31:38Z)
• Neural Operators with Localized Integral and Differential Kernels [77.76991758980003]
  We present a principled approach to operator learning that can capture local features under two frameworks. We prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions.
  arXiv  Detail & Related papers  (2024-02-26T18:59:31Z)
• How many Neurons do we need? A refined Analysis for Shallow Networks trained with Gradient Descent [0.0]
  We analyze the generalization properties of two-layer neural networks in the neural tangent kernel regime. We derive fast rates of convergence that are known to be minimax optimal in the framework of non-parametric regression.
  arXiv  Detail & Related papers  (2023-09-14T22:10:28Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Approximate Bayesian Neural Operators: Uncertainty Quantification for Parametric PDEs [34.179984253109346]
  We provide a mathematically detailed Bayesian formulation of the ''shallow'' (linear) version of neural operators. We then extend this analytic treatment to general deep neural operators using approximate methods from Bayesian deep learning. As a result, our approach is able to identify cases, and provide structured uncertainty estimates, where the neural operator fails to predict well.
  arXiv  Detail & Related papers  (2022-08-02T16:10:27Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• When and why PINNs fail to train: A neural tangent kernel perspective [2.1485350418225244]
  We derive the Neural Tangent Kernel (NTK) of PINNs and prove that, under appropriate conditions, it converges to a deterministic kernel that stays constant during training in the infinite-width limit. We find a remarkable discrepancy in the convergence rate of the different loss components contributing to the total training error. We propose a novel gradient descent algorithm that utilizes the eigenvalues of the NTK to adaptively calibrate the convergence rate of the total training error.
  arXiv  Detail & Related papers  (2020-07-28T23:44:56Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks [87.23360438947114]
  We show that noisy gradient descent with weight decay can still exhibit a " Kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.
  arXiv  Detail & Related papers  (2020-02-10T18:56:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.