Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator

• URL: http://arxiv.org/abs/2510.16816v1
• Date: Sun, 19 Oct 2025 13:03:09 GMT
• Title: Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator
• Abstract summary: We present a novel type of neural operators, Linear Attention Neural Operator (LANO)<n>LANO achieves both scalability and high accuracy by reformulating attention through an agent-based mechanism.<n> Empirically, LANO surpasses current state-of-the-art neural PDE solvers, including Transolver with slice-based softmax attention.
• Score: 2.1595890347557756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural operators offer a powerful data-driven framework for learning mappings between function spaces, in which the transformer-based neural operator architecture faces a fundamental scalability-accuracy trade-off: softmax attention provides excellent fidelity but incurs quadratic complexity $\mathcal{O}(N^2 d)$ in the number of mesh points $N$ and hidden dimension $d$, while linear attention variants reduce cost to $\mathcal{O}(N d^2)$ but often suffer significant accuracy degradation. To address the aforementioned challenge, in this paper, we present a novel type of neural operators, Linear Attention Neural Operator (LANO), which achieves both scalability and high accuracy by reformulating attention through an agent-based mechanism. LANO resolves this dilemma by introducing a compact set of $M$ agent tokens $(M \ll N)$ that mediate global interactions among $N$ tokens. This agent attention mechanism yields an operator layer with linear complexity $\mathcal{O}(MN d)$ while preserving the expressive power of softmax attention. Theoretically, we demonstrate the universal approximation property, thereby demonstrating improved conditioning and stability properties. Empirically, LANO surpasses current state-of-the-art neural PDE solvers, including Transolver with slice-based softmax attention, achieving average $19.5\%$ accuracy improvement across standard benchmarks. By bridging the gap between linear complexity and softmax-level performance, LANO establishes a scalable, high-accuracy foundation for scientific machine learning applications.

Related papers

• Learning High-Dimensional Parity Functions with Product Networks using Gradient Descent [1.6802038598427533]
  High-dimensional parity functions are critical for machine learning, cryptography, and error correction.<n>Standard neural network architectures typically require exponential sample complexity.<n>We show that compact product-based neural architectures combined with data sparsity enable efficient parity learning.<n>This work opens new possibilities for neural arithmetic, structured reasoning, binary neural networks, and machine learning applied to automated protocol discovery.
  arXiv  Detail & Related papers  (2026-05-27T15:26:16Z)
• IKNO: Infinite-order Kernel Neural Operators [57.17011066621087]
  We propose the Infinite-order Kernel Neural Operator (IKNO), which constructs neural operators via infinite-order kernel integrals.<n>We develop two complementary infinite-order neural operator constructions: IKNO-Vanilla, which applies the full- kernel resolvent on the product grid via Kronecker eigendecomposition, and IKNO-TP, an alternative tensor-product operator that composes per-axis resolvents.
  arXiv  Detail & Related papers  (2026-05-21T08:52:36Z)
• Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons [27.855778669417592]
  Cortical neurons are complex, multi-timescale processors wired into recurrent circuits.<n>Machine learning predominantly builds models extremely simple units, a default inherited from early neural-network theory.<n>We introduce the ELM Network, whose recurrent layer is built from Expressive Memory (ELM) neurons.
  arXiv  Detail & Related papers  (2026-05-12T12:29:33Z)
• Hybrid Iterative Neural Low-Regularity Integrator for Nonlinear Dispersive Equations [0.0]
  HIN-LRI is a hybrid framework that augments a classical numerical solver with a neural operator trained to correct the solver's structured truncation error.<n>A lightweight neural network, operating on a low-dimensional latent manifold, learns the residual defect that analytical methods cannot close.<n>Experiments show that HIN-LRI improves accuracy over analytical, splitting methods, and neural PDE surrogates, with stable spatial refinement, effective out-of-distribution transfer, and modest online overhead.
  arXiv  Detail & Related papers  (2026-05-06T12:50:36Z)
• Wiener Chaos Expansion based Neural Operator for Singular Stochastic Partial Differential Equations [58.188204772101756]
  This paper explores how our recently developed Wiener Chaos Expansion (WCE)-based neural operator (NO) can be applied to singular partial differential equations.<n>We leverage feature-wise linear modulation (FiLM) to appropriately capture the dependency between the solution of singular SPDE and its smooth remainder.<n>The resulting WCE-FiLM-NO shows excellent performance on $boldsymbol4$, as measured by relative $L$ loss, out-of-distribution and autocorrelation score.
  arXiv  Detail & Related papers  (2026-03-09T10:50:30Z)
• Redefining Neural Operators in $d+1$ Dimensions [16.36239488071517]
  We redefine neural operators on a new $d+1$ dimensional domain based on Schr"odingerisation method.<n>Within this framework, we implement a Schr"odingerised Kernel Neural Operator (SKNO) aligning better with the $d+1$ dimensional evolution.<n>In experiments, the $d+1$ dimensional evolving designs in our SKNO consistently outperform other baselines.
  arXiv  Detail & Related papers  (2025-05-17T00:15:00Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization [66.03821840425539]
  In this paper, we investigate the training dynamics of $L$-layer neural networks using the tensor gradient program (SGD) framework.<n>We show that SGD enables these networks to learn linearly independent features that substantially deviate from their initial values.<n>This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum.
  arXiv  Detail & Related papers  (2025-03-12T17:33:13Z)
• Near-Optimal Online Learning for Multi-Agent Submodular Coordination: Tight Approximation and Communication Efficiency [52.60557300927007]
  We present a $textbfMA-OSMA$ algorithm to transfer the discrete submodular problem into a continuous optimization.<n>We also introduce a projection-free $textbfMA-OSEA$ algorithm, which effectively utilizes the KL divergence by mixing a uniform distribution.<n>Our algorithms significantly improve the $(frac11+c)$-approximation provided by the state-of-the-art OSG algorithm.
  arXiv  Detail & Related papers  (2025-02-07T15:57:56Z)
• TensorGRaD: Tensor Gradient Robust Decomposition for Memory-Efficient Neural Operator Training [91.8932638236073]
  We introduce textbfTensorGRaD, a novel method that directly addresses the memory challenges associated with large-structured weights.<n>We show that sparseGRaD reduces total memory usage by over $50%$ while maintaining and sometimes even improving accuracy.
  arXiv  Detail & Related papers  (2025-01-04T20:51:51Z)
• Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions [26.025283259518936]
  Rodimus is a new type of attention-based large language models (LLMs)<n>Rodimus employs a data-dependent tempered selection mechanism within a linear attention-based, purely recurrent framework.<n>Rodimus$+$ combines Rodimus with the innovative Sliding Window Shared-Key Attention (SW-SKA) in a hybrid approach.<n>Our experiments demonstrate that Rodimus$+$-1.6B, trained on 1 trillion tokens, achieves superior downstream performance against models trained on more tokens.
  arXiv  Detail & Related papers  (2024-10-09T06:22:36Z)
• Deep Tensor Network [9.910562011343009]
  We introduce the Deep Network, a new architectural framework that reformulates attention by unifying the expressive power of tensor algebra with neural network design.<n>Our approach moves beyond both conventional dot-product attention and subsequent linear-time approximations to capture higher-order statistical dependencies.
  arXiv  Detail & Related papers  (2023-11-18T14:41:33Z)
• MgNO: Efficient Parameterization of Linear Operators via Multigrid [4.096453902709292]
  We introduce MgNO, utilizing multigrid structures to parameterize linear operators between neurons. MgNO exhibits superior ease of training compared to other CNN-based models.
  arXiv  Detail & Related papers  (2023-10-16T13:01:35Z)
• Guaranteed Approximation Bounds for Mixed-Precision Neural Operators [83.64404557466528]
  We build on intuition that neural operator learning inherently induces an approximation error. We show that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
  arXiv  Detail & Related papers  (2023-07-27T17:42:06Z)
• Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representation [59.45669299295436]
  We propose a Monte Carlo PDE solver for training unsupervised neural solvers.<n>We use the PDEs' probabilistic representation, which regards macroscopic phenomena as ensembles of random particles.<n>Our experiments on convection-diffusion, Allen-Cahn, and Navier-Stokes equations demonstrate significant improvements in accuracy and efficiency.
  arXiv  Detail & Related papers  (2023-02-10T08:05:19Z)
• Frame invariance and scalability of neural operators for partial differential equations [5.872676314924041]
  Partial differential equations (PDEs) play a dominant role in the mathematical modeling of many complex dynamical processes. After training, neural operators can provide PDEs solutions significantly faster than traditional PDE solvers.
  arXiv  Detail & Related papers  (2021-12-28T02:36:19Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.