PROSE: Predicting Operators and Symbolic Expressions using Multimodal
Transformers
- URL: http://arxiv.org/abs/2309.16816v1
- Date: Thu, 28 Sep 2023 19:46:07 GMT
- Title: PROSE: Predicting Operators and Symbolic Expressions using Multimodal
Transformers
- Authors: Yuxuan Liu, Zecheng Zhang, Hayden Schaeffer
- Abstract summary: We develop a new neural network framework for predicting differential equations.
By using a transformer structure and a feature fusion approach, our network can simultaneously embed sets of solution operators for various parametric differential equations.
The network is shown to be able to handle noise in the data and errors in the symbolic representation, including noisy numerical values, model misspecification, and erroneous addition or deletion of terms.
- Score: 5.263113622394007
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Approximating nonlinear differential equations using a neural network
provides a robust and efficient tool for various scientific computing tasks,
including real-time predictions, inverse problems, optimal controls, and
surrogate modeling. Previous works have focused on embedding dynamical systems
into networks through two approaches: learning a single solution operator
(i.e., the mapping from input parametrized functions to solutions) or learning
the governing system of equations (i.e., the constitutive model relative to the
state variables). Both of these approaches yield different representations for
the same underlying data or function. Additionally, observing that families of
differential equations often share key characteristics, we seek one network
representation across a wide range of equations. Our method, called Predicting
Operators and Symbolic Expressions (PROSE), learns maps from multimodal inputs
to multimodal outputs, capable of generating both numerical predictions and
mathematical equations. By using a transformer structure and a feature fusion
approach, our network can simultaneously embed sets of solution operators for
various parametric differential equations using a single trained network.
Detailed experiments demonstrate that the network benefits from its multimodal
nature, resulting in improved prediction accuracy and better generalization.
The network is shown to be able to handle noise in the data and errors in the
symbolic representation, including noisy numerical values, model
misspecification, and erroneous addition or deletion of terms. PROSE provides a
new neural network framework for differential equations which allows for more
flexibility and generality in learning operators and governing equations from
data.
Related papers
- Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
We propose to represent neural networks as computational graphs of parameters.
Our approach enables a single model to encode neural computational graphs with diverse architectures.
We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
arXiv Detail & Related papers (2024-03-18T18:01:01Z) - PICL: Physics Informed Contrastive Learning for Partial Differential Equations [7.136205674624813]
We develop a novel contrastive pretraining framework that improves neural operator generalization across multiple governing equations simultaneously.
A combination of physics-informed system evolution and latent-space model output are anchored to input data and used in our distance function.
We find that physics-informed contrastive pretraining improves accuracy for the Fourier Neural Operator in fixed-future and autoregressive rollout tasks for the 1D and 2D Heat, Burgers', and linear advection equations.
arXiv Detail & Related papers (2024-01-29T17:32:22Z) - Physics-Informed Generator-Encoder Adversarial Networks with Latent
Space Matching for Stochastic Differential Equations [14.999611448900822]
We propose a new class of physics-informed neural networks to address the challenges posed by forward, inverse, and mixed problems in differential equations.
Our model consists of two key components: the generator and the encoder, both updated alternately by gradient descent.
In contrast to previous approaches, we employ an indirect matching that operates within the lower-dimensional latent feature space.
arXiv Detail & Related papers (2023-11-03T04:29:49Z) - In-Context Convergence of Transformers [63.04956160537308]
We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent.
For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
arXiv Detail & Related papers (2023-10-08T17:55:33Z) - Structured Radial Basis Function Network: Modelling Diversity for
Multiple Hypotheses Prediction [51.82628081279621]
Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions.
A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems.
It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
arXiv Detail & Related papers (2023-09-02T01:27:53Z) - In-Context Operator Learning with Data Prompts for Differential Equation
Problems [12.61842281581773]
This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON)
ICON simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update.
Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems.
arXiv Detail & Related papers (2023-04-17T05:22:26Z) - A PINN Approach to Symbolic Differential Operator Discovery with Sparse
Data [0.0]
In this work we perform symbolic discovery of differential operators in a situation where there is sparse experimental data.
We modify the PINN approach by adding a neural network that learns a representation of unknown hidden terms in the differential equation.
The algorithm yields both a surrogate solution to the differential equation and a black-box representation of the hidden terms.
arXiv Detail & Related papers (2022-12-09T02:09:37Z) - Equivariant vector field network for many-body system modeling [65.22203086172019]
Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers.
We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
arXiv Detail & Related papers (2021-10-26T14:26:25Z) - Multipole Graph Neural Operator for Parametric Partial Differential
Equations [57.90284928158383]
One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data.
We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity.
Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
arXiv Detail & Related papers (2020-06-16T21:56:22Z) - Liquid Time-constant Networks [117.57116214802504]
We introduce a new class of time-continuous recurrent neural network models.
Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems.
These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
arXiv Detail & Related papers (2020-06-08T09:53:35Z) - Hamiltonian neural networks for solving equations of motion [3.1498833540989413]
We present a Hamiltonian neural network that solves the differential equations that govern dynamical systems.
A symplectic Euler integrator requires two orders more evaluation points than the Hamiltonian network in order to achieve the same order of the numerical error.
arXiv Detail & Related papers (2020-01-29T21:48:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.