The Fourier Spectral Transformer Networks For Efficient and Generalizable Nonlinear PDEs Prediction

• URL: http://arxiv.org/abs/2507.05584v1
• Date: Tue, 08 Jul 2025 01:43:33 GMT
• Title: The Fourier Spectral Transformer Networks For Efficient and Generalizable Nonlinear PDEs Prediction
• Authors: Beibei Li,
• Abstract summary: We use high precision numerical solvers to generate training data and use a Transformer network to model the evolution of the spectral coefficients.<n>The proposed framework generalizes well to unseen data, bringing a promising paradigm for real time prediction and control of complex dynamical systems.
• Score: 1.1272369832509876
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this work we propose a unified Fourier Spectral Transformer network that integrates the strengths of classical spectral methods and attention based neural architectures. By transforming the original PDEs into spectral ordinary differential equations, we use high precision numerical solvers to generate training data and use a Transformer network to model the evolution of the spectral coefficients. We demonstrate the effectiveness of our approach on the two dimensional incompressible Navier-Stokes equations and the one dimensional Burgers' equation. The results show that our spectral Transformer can achieve highly accurate long term predictions even with limited training data, better than traditional numerical methods and machine learning methods in forecasting future flow dynamics. The proposed framework generalizes well to unseen data, bringing a promising paradigm for real time prediction and control of complex dynamical systems.

Related papers

• MixFunn: A Neural Network for Differential Equations with Improved Generalization and Interpretability [0.0]
  MixFunn is a novel neural network architecture designed to solve differential equations with enhanced precision, interpretability, and generalization capability.<n>The architecture comprises two key components: the mixed-function neuron, which integrates multiple parameterized nonlinear functions, and the second-order neuron, which combines a linear transformation of its inputs with a quadratic term to capture cross-combinations of input variables.
  arXiv  Detail & Related papers  (2025-03-28T15:31:15Z)
• Koopman Theory-Inspired Method for Learning Time Advancement Operators in Unstable Flame Front Evolution [0.2812395851874055]
  This study introduces Koopman-inspired Fourier Neural Operators (kFNO) and Convolutional Neural Networks (kCNN) to learn solution advancement operators for flame front instabilities.<n>By transforming data into a high-dimensional latent space, these models achieve more accurate multi-step predictions compared to traditional methods.
  arXiv  Detail & Related papers  (2024-12-11T14:47:19Z)
• From Fourier to Neural ODEs: Flow Matching for Modeling Complex Systems [20.006163951844357]
  We propose a simulation-free framework for training neural ordinary differential equations (NODEs) We employ the Fourier analysis to estimate temporal and potential high-order spatial gradients from noisy observational data. Our approach outperforms state-of-the-art methods in terms of training time, dynamics prediction, and robustness.
  arXiv  Detail & Related papers  (2024-05-19T13:15:23Z)
• Multi-scale Time-stepping of Partial Differential Equations with Transformers [8.430481660019451]
  We develop fast surrogates for Partial Differential Equations (PDEs) Our model achieves similar or better results in predicting the time-evolution of Navier-Stokes equations.
  arXiv  Detail & Related papers  (2023-11-03T20:26:43Z)
• Spectral operator learning for parametric PDEs without data reliance [6.7083321695379885]
  We introduce a novel operator learning-based approach for solving parametric partial differential equations (PDEs) without the need for data harnessing. The proposed framework demonstrates superior performance compared to existing scientific machine learning techniques.
  arXiv  Detail & Related papers  (2023-10-03T12:37:15Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
  We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time. This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
  arXiv  Detail & Related papers  (2022-11-26T01:56:05Z)
• NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer [45.47667026025716]
  We propose a novel, robust and accelerated iteration that relies on two key elements. The convergence and stability of the obtained method, referred to as NAG-GS, are first studied extensively. We show that NAG-arity is competitive with state-the-art methods such as momentum SGD with weight decay and AdamW for the training of machine learning models.
  arXiv  Detail & Related papers  (2022-09-29T16:54:53Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• Combining Differentiable PDE Solvers and Graph Neural Networks for Fluid Flow Prediction [79.81193813215872]
  We develop a hybrid (graph) neural network that combines a traditional graph convolutional network with an embedded differentiable fluid dynamics simulator inside the network itself. We show that we can both generalize well to new situations and benefit from the substantial speedup of neural network CFD predictions.
  arXiv  Detail & Related papers  (2020-07-08T21:23:19Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.