SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs

• URL: http://arxiv.org/abs/2511.18777v1
• Date: Mon, 24 Nov 2025 05:22:28 GMT
• Title: SAOT: An Enhanced Locality-Aware Spectral Transformer for Solving PDEs
• Authors: Chenhong Zhou, Jie Chen, Zaifeng Yang,
• Abstract summary: We investigate incorporating the spatial-frequency localization property of Wavelet transforms into the Transformer architecture.<n>We propose a novel Wavelet Attention (WA) module with linear computational complexity to efficiently learn locality-aware features.<n>We further develop the Spectral Attention Operator Transformer (SAOT), a hybrid spectral Transformer framework that integrates WA's localized focus with the global receptive field of Fourier-based Attention.
• Score: 8.678387342998613
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural operators have shown great potential in solving a family of Partial Differential Equations (PDEs) by modeling the mappings between input and output functions. Fourier Neural Operator (FNO) implements global convolutions via parameterizing the integral operators in Fourier space. However, it often results in over-smoothing solutions and fails to capture local details and high-frequency components. To address these limitations, we investigate incorporating the spatial-frequency localization property of Wavelet transforms into the Transformer architecture. We propose a novel Wavelet Attention (WA) module with linear computational complexity to efficiently learn locality-aware features. Building upon WA, we further develop the Spectral Attention Operator Transformer (SAOT), a hybrid spectral Transformer framework that integrates WA's localized focus with the global receptive field of Fourier-based Attention (FA) through a gated fusion block. Experimental results demonstrate that WA significantly mitigates the limitations of FA and outperforms existing Wavelet-based neural operators by a large margin. By integrating the locality-aware and global spectral representations, SAOT achieves state-of-the-art performance on six operator learning benchmarks and exhibits strong discretization-invariant ability.

Related papers

• SpectraKAN: Conditioning Spectral Operators [21.190440188964452]
  We introduce SpectraKAN, a neural operator that conditions the spectral operator on the input itself, turning spectral into an input-conditioned integral operator.<n>This is achieved by extracting a compact global representation from static-temporal history and using it to modulate a multi-scale trunk via single-query cross-attention.<n>Across diverse PDE benchmarks, SpectraKAN achieves state-of-the-art performance, reducing RMSE by up to 49% over strong baselines, with particularly large gains on challenging-temporal prediction tasks.
  arXiv  Detail & Related papers  (2026-02-05T01:30:25Z)
• Fourier-RWKV: A Multi-State Perception Network for Efficient Image Dehazing [26.57698394898644]
  We propose a novel dehazing framework based on a Multi-State Perception paradigm.<n>Fourier-RWKV delivers state-of-the-art performance across diverse haze scenarios.
  arXiv  Detail & Related papers  (2025-12-09T01:35:56Z)
• Feature-Modulated UFNO for Improved Prediction of Multiphase Flow in Porous Media [0.39146761527401425]
  We introduce UFNO-FiLM, an enhanced architecture that incorporates two key innovations.<n>First, we decouple scalar inputs from spatial features using a Feature-wise Linear Modulation layer.<n>Second, we employ a spatially weighted loss function that prioritizes learning in critical regions.
  arXiv  Detail & Related papers  (2025-11-25T17:44:28Z)
• Wavelet-Guided Dual-Frequency Encoding for Remote Sensing Change Detection [67.84730634802204]
  Change detection in remote sensing imagery plays a vital role in various engineering applications, such as natural disaster monitoring, urban expansion tracking, and infrastructure management.<n>Most existing methods still rely on spatial-domain modeling, where the limited diversity of feature representations hinders the detection of subtle change regions.<n>We observe that frequency-domain feature modeling particularly in the wavelet domain amplify fine-grained differences in frequency components, enhancing the perception of edge changes that are challenging to capture in the spatial domain.
  arXiv  Detail & Related papers  (2025-08-07T11:14:16Z)
• GITO: Graph-Informed Transformer Operator for Learning Complex Partial Differential Equations [0.0]
  We present a novel graph-informed transformer operator (GITO) architecture for learning complex partial differential equation systems.<n>GITO consists of two main modules: a hybrid graph transformer (HGT) and a transformer neural operator (TNO)<n> Empirical results on benchmark PDE tasks demonstrate that GITO outperforms existing transformer-based neural operators.
  arXiv  Detail & Related papers  (2025-06-16T18:35:45Z)
• STAF: Sinusoidal Trainable Activation Functions for Implicit Neural Representation [7.2888019138115245]
  Implicit Neural Representations (INRs) have emerged as a powerful framework for modeling continuous signals.<n>The spectral bias of ReLU-based networks is a well-established limitation, restricting their ability to capture fine-grained details in target signals.<n>We introduce Sinusoidal Trainable Functions Activation (STAF)<n>STAF inherently modulates its frequency components, allowing for self-adaptive spectral learning.
  arXiv  Detail & Related papers  (2025-02-02T18:29:33Z)
• DimINO: Dimension-Informed Neural Operator Learning [41.37905663176428]
  DimINO is a framework inspired by dimensional analysis.<n>It can be seamlessly integrated into existing neural operator architectures.<n>It achieves up to 76.3% performance gain on PDE datasets.
  arXiv  Detail & Related papers  (2024-10-08T10:48:50Z)
• Spectral Enhanced Rectangle Transformer for Hyperspectral Image Denoising [64.11157141177208]
  We propose a spectral enhanced rectangle Transformer to model the spatial and spectral correlation in hyperspectral images. For the former, we exploit the rectangle self-attention horizontally and vertically to capture the non-local similarity in the spatial domain. For the latter, we design a spectral enhancement module that is capable of extracting global underlying low-rank property of spatial-spectral cubes to suppress noise.
  arXiv  Detail & Related papers  (2023-04-03T09:42:13Z)
• Multiscale Attention via Wavelet Neural Operators for Vision Transformers [0.0]
  Transformers have achieved widespread success in computer vision. At their heart, there is a Self-Attention (SA) mechanism. Standard SA mechanism has quadratic complexity with the sequence length, which impedes its utility to long sequences appearing in high resolution vision. We introduce a Multiscale Wavelet Attention (MWA) by leveraging wavelet neural operators which incurs linear complexity in the sequence size.
  arXiv  Detail & Related papers  (2023-03-22T09:06:07Z)
• Deep Fourier Up-Sampling [100.59885545206744]
  Up-sampling in the Fourier domain is more challenging as it does not follow such a local property. We propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues.
  arXiv  Detail & Related papers  (2022-10-11T06:17:31Z)
• FAMLP: A Frequency-Aware MLP-Like Architecture For Domain Generalization [73.41395947275473]
  We propose a novel frequency-aware architecture, in which the domain-specific features are filtered out in the transformed frequency domain. Experiments on three benchmarks demonstrate significant performance, outperforming the state-of-the-art methods by a margin of 3%, 4% and 9%, respectively.
  arXiv  Detail & Related papers  (2022-03-24T07:26:29Z)
• Factorized Fourier Neural Operators [77.47313102926017]
  The Factorized Fourier Neural Operator (F-FNO) is a learning-based method for simulating partial differential equations. We show that our model maintains an error rate of 2% while still running an order of magnitude faster than a numerical solver.
  arXiv  Detail & Related papers  (2021-11-27T03:34:13Z)
• Beyond Self Attention: A Subquadratic Fourier Wavelet Transformer with Multi Modal Fusion [0.0]
  We revisit the use of spectral techniques to replace the attention mechanism in Transformers.<n>We present a comprehensive and novel reformulation of this technique in next generation transformer models.
  arXiv  Detail & Related papers  (2021-11-25T18:03:41Z)
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers [55.90468016961356]
  We propose an efficient token mixer that learns to mix in the Fourier domain. AFNO is based on a principled foundation of operator learning. It can handle a sequence size of 65k and outperforms other efficient self-attention mechanisms.
  arXiv  Detail & Related papers  (2021-11-24T05:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.