How important are specialized transforms in Neural Operators?

• URL: http://arxiv.org/abs/2308.09293v1
• Date: Fri, 18 Aug 2023 04:35:13 GMT
• Title: How important are specialized transforms in Neural Operators?
• Authors: Ritam Majumdar, Shirish Karande, Lovekesh Vig
• Abstract summary: We investigate the importance of the transform layers to the reported success of transform based neural operators. Surprisingly, we observe that linear layers suffice to provide performance comparable to the best-known transform-based layers and seem to do so with a compute time advantage as well.
• Score: 9.809251473887594
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Simulating physical systems using Partial Differential Equations (PDEs) has become an indispensible part of modern industrial process optimization. Traditionally, numerical solvers have been used to solve the associated PDEs, however recently Transform-based Neural Operators such as the Fourier Neural Operator and Wavelet Neural Operator have received a lot of attention for their potential to provide fast solutions for systems of PDEs. In this work, we investigate the importance of the transform layers to the reported success of transform based neural operators. In particular, we record the cost in terms of performance, if all the transform layers are replaced by learnable linear layers. Surprisingly, we observe that linear layers suffice to provide performance comparable to the best-known transform-based layers and seem to do so with a compute time advantage as well. We believe that this observation can have significant implications for future work on Neural Operators, and might point to other sources of efficiencies for these architectures.

Related papers

• Hilbert Neural Operator: Operator Learning in the Analytic Signal Domain [0.0]
  We introduce the textbfHilbert Neural Operator (HNO), a new neural operator architecture to address some advantages.<n>HNO operates by first mapping the input signal to its analytic representation via the Hilbert transform.<n>We hypothesize that this architecture enables HNO to model operators more effectively for causal, phase-sensitive, and non-stationary systems.
  arXiv  Detail & Related papers  (2025-08-06T21:12:15Z)
• Principled Approaches for Extending Neural Architectures to Function Spaces for Operator Learning [78.88684753303794]
  Deep learning has predominantly advanced through applications in computer vision and natural language processing.<n>Neural operators are a principled way to generalize neural networks to mappings between function spaces.<n>This paper identifies and distills the key principles for constructing practical implementations of mappings between infinite-dimensional function spaces.
  arXiv  Detail & Related papers  (2025-06-12T17:59:31Z)
• Neural Interpretable PDEs: Harmonizing Fourier Insights with Attention for Scalable and Interpretable Physics Discovery [15.29112632863168]
  We introduce Neural Interpretable PDEs (NIPS), a novel neural operator architecture that builds upon and enhances Nonlocal Attention Operators (NAO)<n>NIPS employs a linear attention mechanism to enable scalable learning and integrates a learnable kernel network that acts as a channel-independent convolution in Fourier space.<n> Empirical evaluations demonstrate that NIPS consistently surpasses NAO and other baselines across diverse benchmarks.
  arXiv  Detail & Related papers  (2025-05-29T05:18:30Z)
• Exploring Kernel Transformations for Implicit Neural Representations [57.2225355625268]
  Implicit neural representations (INRs) leverage neural networks to represent signals by mapping coordinates to their corresponding attributes. This work pioneers the exploration of the effect of kernel transformation of input/output while keeping the model itself unchanged. A byproduct of our findings is a simple yet effective method that combines scale and shift to significantly boost INR with negligible overhead.
  arXiv  Detail & Related papers  (2025-04-07T04:43:50Z)
• DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning [63.5925701087252]
  We introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets.
  arXiv  Detail & Related papers  (2024-10-08T10:48:50Z)
• Component Fourier Neural Operator for Singularly Perturbed Differential Equations [3.9482103923304877]
  Solving Singularly Perturbed Differential Equations (SPDEs) poses computational challenges arising from the rapid transitions in their solutions within thin regions. In this manuscript, we introduce Component Fourier Neural Operator (ComFNO), an innovative operator learning method that builds upon Fourier Neural Operator (FNO) Our approach is not limited to FNO and can be applied to other neural network frameworks, such as Deep Operator Network (DeepONet)
  arXiv  Detail & Related papers  (2024-09-07T09:40:51Z)
• DeltaPhi: Learning Physical Trajectory Residual for PDE Solving [54.13671100638092]
  We propose and formulate the Physical Trajectory Residual Learning (DeltaPhi) We learn the surrogate model for the residual operator mapping based on existing neural operator networks. We conclude that, compared to direct learning, physical residual learning is preferred for PDE solving.
  arXiv  Detail & Related papers  (2024-06-14T07:45:07Z)
• Neural Operators with Localized Integral and Differential Kernels [77.76991758980003]
  We present a principled approach to operator learning that can capture local features under two frameworks. We prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions.
  arXiv  Detail & Related papers  (2024-02-26T18:59:31Z)
• PICL: Physics Informed Contrastive Learning for Partial Differential Equations [7.136205674624813]
  We develop a novel contrastive pretraining framework that improves neural operator generalization across multiple governing equations simultaneously. A combination of physics-informed system evolution and latent-space model output are anchored to input data and used in our distance function. We find that physics-informed contrastive pretraining improves accuracy for the Fourier Neural Operator in fixed-future and autoregressive rollout tasks for the 1D and 2D Heat, Burgers', and linear advection equations.
  arXiv  Detail & Related papers  (2024-01-29T17:32:22Z)
• GIT-Net: Generalized Integral Transform for Operator Learning [58.13313857603536]
  This article introduces GIT-Net, a deep neural network architecture for approximating Partial Differential Equation (PDE) operators. GIT-Net harnesses the fact that differential operators commonly used for defining PDEs can often be represented parsimoniously when expressed in specialized functional bases. Numerical experiments demonstrate that GIT-Net is a competitive neural network operator, exhibiting small test errors and low evaluations across a range of PDE problems.
  arXiv  Detail & Related papers  (2023-12-05T03:03:54Z)
• Neural Operators for Accelerating Scientific Simulations and Design [85.89660065887956]
  An AI framework, known as Neural Operators, presents a principled framework for learning mappings between functions defined on continuous domains. Neural Operators can augment or even replace existing simulators in many applications, such as computational fluid dynamics, weather forecasting, and material modeling.
  arXiv  Detail & Related papers  (2023-09-27T00:12:07Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• An unsupervised latent/output physics-informed convolutional-LSTM network for solving partial differential equations using peridynamic differential operator [0.0]
  Unsupervised convolutional Neural Network (NN) architecture with nonlocal interactions for solving Partial Differential Equations (PDEs) PDDO is employed as a convolutional filter for evaluating derivatives the field variable. NN captures the time-dynamics in smaller latent space through encoder-decoder layers with a Convolutional Long-short Term Memory (ConvLSTM) layer between them.
  arXiv  Detail & Related papers  (2022-10-21T18:09:23Z)
• Frame invariance and scalability of neural operators for partial differential equations [5.872676314924041]
  Partial differential equations (PDEs) play a dominant role in the mathematical modeling of many complex dynamical processes. After training, neural operators can provide PDEs solutions significantly faster than traditional PDE solvers.
  arXiv  Detail & Related papers  (2021-12-28T02:36:19Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.