Incremental Spatial and Spectral Learning of Neural Operators for
Solving Large-Scale PDEs
- URL: http://arxiv.org/abs/2211.15188v4
- Date: Tue, 5 Mar 2024 04:42:21 GMT
- Title: Incremental Spatial and Spectral Learning of Neural Operators for
Solving Large-Scale PDEs
- Authors: Robert Joseph George, Jiawei Zhao, Jean Kossaifi, Zongyi Li, Anima
Anandkumar
- Abstract summary: We introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases the number of frequency modes used by the model.
We show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets.
Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.
- Score: 86.35471039808023
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Fourier Neural Operators (FNO) offer a principled approach to solving
challenging partial differential equations (PDE) such as turbulent flows. At
the core of FNO is a spectral layer that leverages a discretization-convergent
representation in the Fourier domain, and learns weights over a fixed set of
frequencies. However, training FNO presents two significant challenges,
particularly in large-scale, high-resolution applications: (i) Computing
Fourier transform on high-resolution inputs is computationally intensive but
necessary since fine-scale details are needed for solving many PDEs, such as
fluid flows, (ii) selecting the relevant set of frequencies in the spectral
layers is challenging, and too many modes can lead to overfitting, while too
few can lead to underfitting. To address these issues, we introduce the
Incremental Fourier Neural Operator (iFNO), which progressively increases both
the number of frequency modes used by the model as well as the resolution of
the training data. We empirically show that iFNO reduces total training time
while maintaining or improving generalization performance across various
datasets. Our method demonstrates a 10% lower testing error, using 20% fewer
frequency modes compared to the existing Fourier Neural Operator, while also
achieving a 30% faster training.
Related papers
- Physics-embedded Fourier Neural Network for Partial Differential Equations [35.41134465442465]
We introduce Physics-embedded Fourier Neural Networks (PeFNN) with flexible and explainable error.
PeFNN is designed to enforce momentum conservation and yields interpretable nonlinear expressions.
We demonstrate its outstanding performance for challenging real-world applications such as large-scale flood simulations.
arXiv Detail & Related papers (2024-07-15T18:30:39Z) - Spectral-Refiner: Fine-Tuning of Accurate Spatiotemporal Neural Operator for Turbulent Flows [6.961408873053586]
We propose a new Stemporal Neural Operator (SFNO) that learns maps between Bochner spaces, and a new learning framework to address these issues.
This new paradigm leverages wisdom from traditional numerical PDE theory and techniques to refine the pipeline of commonly adopted end-to-end neural operator training and evaluations.
Numerical experiments on commonly used benchmarks for the 2D NSE demonstrate significant improvements in both computational efficiency and accuracy, compared to end-to-end evaluation and traditional numerical PDE solvers.
arXiv Detail & Related papers (2024-05-27T14:33:06Z) - Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs [93.82811501035569]
We introduce a new data efficient and highly parallelizable operator learning approach with reduced memory requirement and better generalization.
MG-TFNO scales to large resolutions by leveraging local and global structures of full-scale, real-world phenomena.
We demonstrate superior performance on the turbulent Navier-Stokes equations where we achieve less than half the error with over 150x compression.
arXiv Detail & Related papers (2023-09-29T20:18:52Z) - Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time.
This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
arXiv Detail & Related papers (2022-11-26T01:56:05Z) - Solving Seismic Wave Equations on Variable Velocity Models with Fourier
Neural Operator [3.2307366446033945]
We propose a new framework paralleled Fourier neural operator (PFNO) for efficiently training the FNO-based solver.
Numerical experiments demonstrate the high accuracy of both FNO and PFNO with complicated velocity models.
PFNO admits higher computational efficiency on large-scale testing datasets, compared with the traditional finite-difference method.
arXiv Detail & Related papers (2022-09-25T22:25:57Z) - Momentum Diminishes the Effect of Spectral Bias in Physics-Informed
Neural Networks [72.09574528342732]
Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs)
They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias.
In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
arXiv Detail & Related papers (2022-06-29T19:03:10Z) - Factorized Fourier Neural Operators [77.47313102926017]
The Factorized Fourier Neural Operator (F-FNO) is a learning-based method for simulating partial differential equations.
We show that our model maintains an error rate of 2% while still running an order of magnitude faster than a numerical solver.
arXiv Detail & Related papers (2021-11-27T03:34:13Z) - Learning Frequency Domain Approximation for Binary Neural Networks [68.79904499480025]
We propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs.
The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy.
arXiv Detail & Related papers (2021-03-01T08:25:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.