Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective
- URL: http://arxiv.org/abs/2310.06379v3
- Date: Thu, 26 Sep 2024 07:05:47 GMT
- Title: Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective
- Authors: Takeshi Koshizuka, Masahiro Fujisawa, Yusuke Tanaka, Issei Sato,
- Abstract summary: We explore the expressivity and trainability of the Fourier Neural Operator (FNO)
We examine the ordered-chaos phase transition of the network based on the weight distribution.
We find a connection between expressivity and trainability: the ordered and chaotic phases correspond to regions of vanishing and exploding gradients.
- Score: 31.030338985431722
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we explores the expressivity and trainability of the Fourier Neural Operator (FNO). We establish a mean-field theory for the FNO, analyzing the behavior of the random FNO from an edge of chaos perspective. Our investigation into the expressivity of a random FNO involves examining the ordered-chaos phase transition of the network based on the weight distribution. This phase transition demonstrates characteristics unique to the FNO, induced by mode truncation, while also showcasing similarities to those of densely connected networks. Furthermore, we identify a connection between expressivity and trainability: the ordered and chaotic phases correspond to regions of vanishing and exploding gradients, respectively. This finding provides a practical prerequisite for the stable training of the FNO. Our experimental results corroborate our theoretical findings.
Related papers
- Learning S-Matrix Phases with Neural Operators [0.0]
We study the relation between the modulus and phase of amplitudes in $2to 2$ elastic scattering at fixed energies.
We do not employ the integral relation imposed by unitarity, but instead train FNOs to discover it from many samples of amplitudes with finite partial wave expansions.
arXiv Detail & Related papers (2024-04-22T19:46:07Z) - Phase Matching for Out-of-Distribution Generalization [9.786356781007122]
This paper is dedicated to clarifying the relationships between Domain Generalization (DG) and the frequency components.
We propose a Phase Matching approach, termed PhaMa, to address DG problems.
Experiments on multiple benchmarks demonstrate that our proposed method achieves state-of-the-art performance.
arXiv Detail & Related papers (2023-07-24T08:51:49Z) - D4FT: A Deep Learning Approach to Kohn-Sham Density Functional Theory [79.50644650795012]
We propose a deep learning approach to solve Kohn-Sham Density Functional Theory (KS-DFT)
We prove that such an approach has the same expressivity as the SCF method, yet reduces the computational complexity.
In addition, we show that our approach enables us to explore more complex neural-based wave functions.
arXiv Detail & Related papers (2023-03-01T10:38:10Z) - Learning Discretized Neural Networks under Ricci Flow [51.36292559262042]
We study Discretized Neural Networks (DNNs) composed of low-precision weights and activations.
DNNs suffer from either infinite or zero gradients due to the non-differentiable discrete function during training.
arXiv Detail & Related papers (2023-02-07T10:51:53Z) - Stochastic Gradient Descent-Induced Drift of Representation in a
Two-Layer Neural Network [0.0]
Despite being observed in the brain and in artificial networks, the mechanisms of drift and its implications are not fully understood.
Motivated by recent experimental findings of stimulus-dependent drift in the piriform cortex, we use theory and simulations to study this phenomenon in a two-layer linear feedforward network.
arXiv Detail & Related papers (2023-02-06T04:56:05Z) - Momentum Diminishes the Effect of Spectral Bias in Physics-Informed
Neural Networks [72.09574528342732]
Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs)
They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias.
In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
arXiv Detail & Related papers (2022-06-29T19:03:10Z) - Discretely Indexed Flows [1.0079626733116611]
We propose Discretely Indexed flows (DIF) as a new tool for solving variational estimation problems.
DIF are built as an extension of Normalizing Flows (NF), in which the deterministic transport becomes discretely indexed.
They benefit from both a tractable density as well as a straightforward sampling scheme, and can thus be used for the dual problems of Variational Inference (VI) and of Variational density estimation (VDE)
arXiv Detail & Related papers (2022-04-04T10:13:43Z) - Convex Analysis of the Mean Field Langevin Dynamics [49.66486092259375]
convergence rate analysis of the mean field Langevin dynamics is presented.
$p_q$ associated with the dynamics allows us to develop a convergence theory parallel to classical results in convex optimization.
arXiv Detail & Related papers (2022-01-25T17:13:56Z) - Factorized Fourier Neural Operators [77.47313102926017]
The Factorized Fourier Neural Operator (F-FNO) is a learning-based method for simulating partial differential equations.
We show that our model maintains an error rate of 2% while still running an order of magnitude faster than a numerical solver.
arXiv Detail & Related papers (2021-11-27T03:34:13Z) - Stability of Neural Networks on Manifolds to Relative Perturbations [118.84154142918214]
Graph Neural Networks (GNNs) show impressive performance in many practical scenarios.
GNNs can scale well on large size graphs, but this is contradicted by the fact that existing stability bounds grow with the number of nodes.
arXiv Detail & Related papers (2021-10-10T04:37:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.