Neural Integral Equations

• URL: http://arxiv.org/abs/2209.15190v5
• Date: Tue, 10 Sep 2024 21:18:35 GMT
• Title: Neural Integral Equations
• Authors: Emanuele Zappala, Antonio Henrique de Oliveira Fonseca, Josue Ortega Caro, Andrew Henry Moberly, Michael James Higley, Jessica Cardin, David van Dijk,
• Abstract summary: We introduce a method for learning unknown integral operators from data using an IE solver. We also present Attentional Neural Integral Equations (ANIE), which replaces the integral with self-attention.
• Score: 3.087238735145305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Nonlinear operators with long distance spatiotemporal dependencies are fundamental in modeling complex systems across sciences, yet learning these nonlocal operators remains challenging in machine learning. Integral equations (IEs), which model such nonlocal systems, have wide ranging applications in physics, chemistry, biology, and engineering. We introduce Neural Integral Equations (NIE), a method for learning unknown integral operators from data using an IE solver. To improve scalability and model capacity, we also present Attentional Neural Integral Equations (ANIE), which replaces the integral with self-attention. Both models are grounded in the theory of second kind integral equations, where the indeterminate appears both inside and outside the integral operator. We provide theoretical analysis showing how self-attention can approximate integral operators under mild regularity assumptions, further deepening previously reported connections between transformers and integration, and deriving corresponding approximation results for integral operators. Through numerical benchmarks on synthetic and real world data, including Lotka-Volterra, Navier-Stokes, and Burgers' equations, as well as brain dynamics and integral equations, we showcase the models' capabilities and their ability to derive interpretable dynamics embeddings. Our experiments demonstrate that ANIE outperforms existing methods, especially for longer time intervals and higher dimensional problems. Our work addresses a critical gap in machine learning for nonlocal operators and offers a powerful tool for studying unknown complex systems with long range dependencies.

Related papers

• DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning [63.5925701087252]
  We introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets.
  arXiv  Detail & Related papers  (2024-10-08T10:48:50Z)
• PICL: Physics Informed Contrastive Learning for Partial Differential Equations [7.136205674624813]
  We develop a novel contrastive pretraining framework that improves neural operator generalization across multiple governing equations simultaneously. A combination of physics-informed system evolution and latent-space model output are anchored to input data and used in our distance function. We find that physics-informed contrastive pretraining improves accuracy for the Fourier Neural Operator in fixed-future and autoregressive rollout tasks for the 1D and 2D Heat, Burgers', and linear advection equations.
  arXiv  Detail & Related papers  (2024-01-29T17:32:22Z)
• Spectral methods for Neural Integral Equations [0.6993026261767287]
  We introduce a framework for neural integral equations based on spectral methods. We show various theoretical guarantees regarding the approximation capabilities of the model. We provide numerical experiments to demonstrate the practical effectiveness of the resulting model.
  arXiv  Detail & Related papers  (2023-12-09T19:42:36Z)
• Learning nonlinear integral operators via Recurrent Neural Networks and its application in solving Integro-Differential Equations [4.011446845089061]
  We learn and represent nonlinear integral operators that appear in nonlinear integro-differential equations (IDEs) The LSTM-RNN representation of the nonlinear integral operator allows us to turn a system of nonlinear integro-differential equations into a system of ordinary differential equations. We show how this methodology can effectively solve the Dyson's equation for quantum many-body systems.
  arXiv  Detail & Related papers  (2023-10-13T22:57:46Z)
• CoNO: Complex Neural Operator for Continuous Dynamical Systems [10.326780211731263]
  We introduce a Complex Neural Operator (CoNO) that parameterizes the integral kernel in the complex fractional Fourier domain. We show that the model effectively captures the underlying partial differential equation with a single complex fractional Fourier transform.
  arXiv  Detail & Related papers  (2023-10-03T14:38:12Z)
• Neural Operators for Accelerating Scientific Simulations and Design [85.89660065887956]
  An AI framework, known as Neural Operators, presents a principled framework for learning mappings between functions defined on continuous domains. Neural Operators can augment or even replace existing simulators in many applications, such as computational fluid dynamics, weather forecasting, and material modeling.
  arXiv  Detail & Related papers  (2023-09-27T00:12:07Z)
• On Robust Numerical Solver for ODE via Self-Attention Mechanism [82.95493796476767]
  We explore training efficient and robust AI-enhanced numerical solvers with a small data size by mitigating intrinsic noise disturbances. We first analyze the ability of the self-attention mechanism to regulate noise in supervised learning and then propose a simple-yet-effective numerical solver, Attr, which introduces an additive self-attention mechanism to the numerical solution of differential equations.
  arXiv  Detail & Related papers  (2023-02-05T01:39:21Z)
• Semi-supervised Learning of Partial Differential Operators and Dynamical Flows [68.77595310155365]
  We present a novel method that combines a hyper-network solver with a Fourier Neural Operator architecture. We test our method on various time evolution PDEs, including nonlinear fluid flows in one, two, and three spatial dimensions. The results show that the new method improves the learning accuracy at the time point of supervision point, and is able to interpolate and the solutions to any intermediate time.
  arXiv  Detail & Related papers  (2022-07-28T19:59:14Z)
• Neural Integro-Differential Equations [2.001149416674759]
  Integro-Differential Equations (IDEs) are generalizations of differential equations that comprise both an integral and a differential component. NIDE is a framework that models ordinary and integral components ofIDEs using neural networks. We show that NIDE can decompose dynamics into its Markovian and non-Markovian constituents.
  arXiv  Detail & Related papers  (2022-06-28T20:39:35Z)
• Neural Laplace: Learning diverse classes of differential equations in the Laplace domain [86.52703093858631]
  We propose a unified framework for learning diverse classes of differential equations (DEs) including all the aforementioned ones. Instead of modelling the dynamics in the time domain, we model it in the Laplace domain, where the history-dependencies and discontinuities in time can be represented as summations of complex exponentials. In the experiments, Neural Laplace shows superior performance in modelling and extrapolating the trajectories of diverse classes of DEs.
  arXiv  Detail & Related papers  (2022-06-10T02:14:59Z)
• Fourier Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  We formulate a new neural operator by parameterizing the integral kernel directly in Fourier space. We perform experiments on Burgers' equation, Darcy flow, and Navier-Stokes equation. It is up to three orders of magnitude faster compared to traditional PDE solvers.
  arXiv  Detail & Related papers  (2020-10-18T00:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.