In-Context Operator Learning with Data Prompts for Differential Equation Problems

• URL: http://arxiv.org/abs/2304.07993v3
• Date: Tue, 19 Sep 2023 20:00:39 GMT
• Title: In-Context Operator Learning with Data Prompts for Differential Equation Problems
• Authors: Liu Yang, Siting Liu, Tingwei Meng, Stanley J. Osher
• Abstract summary: This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON) ICON simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems.
• Score: 12.61842281581773
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON), to simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Existing methods are limited to using a neural network to approximate a specific equation solution or a specific operator, requiring retraining when switching to a new problem with different equations. By training a single neural network as an operator learner, we can not only get rid of retraining (even fine-tuning) the neural network for new problems, but also leverage the commonalities shared across operators so that only a few demos in the prompt are needed when learning a new operator. Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems, including forward and inverse problems of ordinary differential equations (ODEs), partial differential equations (PDEs), and mean-field control (MFC) problems, and also show that it can generalize its learning capability to operators beyond the training distribution.

Related papers

• PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers [5.263113622394007]
  We develop a new neural network framework for predicting differential equations. By using a transformer structure and a feature fusion approach, our network can simultaneously embed sets of solution operators for various parametric differential equations. The network is shown to be able to handle noise in the data and errors in the symbolic representation, including noisy numerical values, model misspecification, and erroneous addition or deletion of terms.
  arXiv  Detail & Related papers  (2023-09-28T19:46:07Z)
• Hyena Neural Operator for Partial Differential Equations [9.438207505148947]
  Recent advances in deep learning have provided a new approach to solving partial differential equations that involves the use of neural operators. This study utilizes a neural operator called Hyena, which employs a long convolutional filter that is parameterized by a multilayer perceptron. Our findings indicate Hyena can serve as an efficient and accurate model for partial learning differential equations solution operator.
  arXiv  Detail & Related papers  (2023-06-28T19:45:45Z)
• Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning [11.11883703395469]
  This research offers a fresh take on neural operators with a framework Representation equivalent Neural Operators (ReNO) At its core is the concept of operator aliasing, which measures inconsistency between neural operators and their discrete representations. Our findings detail how aliasing introduces errors when handling different discretizations and grids and loss of crucial continuous structures.
  arXiv  Detail & Related papers  (2023-05-31T14:45:34Z)
• Physics-Informed Neural Operators [3.9181541460605116]
  Neural networks can approximate general nonlinear operators, represented either explicitly by a combination of mathematical operators, e.g., in an advection-diffusion-reaction partial differential equation. The first neural operator was the Deep Operator Network (DeepONet), proposed in 2019 based on rigorous approximation theory. For black box systems, training of neural operators is data-driven only but if the governing equations are known they can be incorporated into the loss function during training to develop physics-informed neural operators.
  arXiv  Detail & Related papers  (2022-07-08T12:29:09Z)
• MIONet: Learning multiple-input operators via tensor product [2.5426761219054312]
  We study the operator regression via neural networks for multiple-input operators defined on the product of Banach spaces. Based on our theory and a low-rank approximation, we propose a novel neural operator, MIONet, to learn multiple-input operators.
  arXiv  Detail & Related papers  (2022-02-12T20:37:04Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Characterizing possible failure modes in physics-informed neural networks [55.83255669840384]
  Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena even for simple PDEs. We show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize.
  arXiv  Detail & Related papers  (2021-09-02T16:06:45Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• Conditional physics informed neural networks [85.48030573849712]
  We introduce conditional PINNs (physics informed neural networks) for estimating the solution of classes of eigenvalue problems. We show that a single deep neural network can learn the solution of partial differential equations for an entire class of problems.
  arXiv  Detail & Related papers  (2021-04-06T18:29:14Z)
• Fourier Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  We formulate a new neural operator by parameterizing the integral kernel directly in Fourier space. We perform experiments on Burgers' equation, Darcy flow, and Navier-Stokes equation. It is up to three orders of magnitude faster compared to traditional PDE solvers.
  arXiv  Detail & Related papers  (2020-10-18T00:34:21Z)
• Self-Organized Operational Neural Networks with Generative Neurons [87.32169414230822]
  ONNs are heterogenous networks with a generalized neuron model that can encapsulate any set of non-linear operators. We propose Self-organized ONNs (Self-ONNs) with generative neurons that have the ability to adapt (optimize) the nodal operator of each connection.
  arXiv  Detail & Related papers  (2020-04-24T14:37:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.