PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers

• URL: http://arxiv.org/abs/2309.16816v1
• Date: Thu, 28 Sep 2023 19:46:07 GMT
• Title: PROSE: Predicting Operators and Symbolic Expressions using Multimodal Transformers
• Authors: Yuxuan Liu, Zecheng Zhang, Hayden Schaeffer
• Abstract summary: We develop a new neural network framework for predicting differential equations. By using a transformer structure and a feature fusion approach, our network can simultaneously embed sets of solution operators for various parametric differential equations. The network is shown to be able to handle noise in the data and errors in the symbolic representation, including noisy numerical values, model misspecification, and erroneous addition or deletion of terms.
• Score: 5.263113622394007
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Approximating nonlinear differential equations using a neural network provides a robust and efficient tool for various scientific computing tasks, including real-time predictions, inverse problems, optimal controls, and surrogate modeling. Previous works have focused on embedding dynamical systems into networks through two approaches: learning a single solution operator (i.e., the mapping from input parametrized functions to solutions) or learning the governing system of equations (i.e., the constitutive model relative to the state variables). Both of these approaches yield different representations for the same underlying data or function. Additionally, observing that families of differential equations often share key characteristics, we seek one network representation across a wide range of equations. Our method, called Predicting Operators and Symbolic Expressions (PROSE), learns maps from multimodal inputs to multimodal outputs, capable of generating both numerical predictions and mathematical equations. By using a transformer structure and a feature fusion approach, our network can simultaneously embed sets of solution operators for various parametric differential equations using a single trained network. Detailed experiments demonstrate that the network benefits from its multimodal nature, resulting in improved prediction accuracy and better generalization. The network is shown to be able to handle noise in the data and errors in the symbolic representation, including noisy numerical values, model misspecification, and erroneous addition or deletion of terms. PROSE provides a new neural network framework for differential equations which allows for more flexibility and generality in learning operators and governing equations from data.

Related papers

• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• PICL: Physics Informed Contrastive Learning for Partial Differential Equations [7.136205674624813]
  We develop a novel contrastive pretraining framework that improves neural operator generalization across multiple governing equations simultaneously. A combination of physics-informed system evolution and latent-space model output are anchored to input data and used in our distance function. We find that physics-informed contrastive pretraining improves accuracy for the Fourier Neural Operator in fixed-future and autoregressive rollout tasks for the 1D and 2D Heat, Burgers', and linear advection equations.
  arXiv  Detail & Related papers  (2024-01-29T17:32:22Z)
• Physics-Informed Generator-Encoder Adversarial Networks with Latent Space Matching for Stochastic Differential Equations [14.999611448900822]
  We propose a new class of physics-informed neural networks to address the challenges posed by forward, inverse, and mixed problems in differential equations. Our model consists of two key components: the generator and the encoder, both updated alternately by gradient descent. In contrast to previous approaches, we employ an indirect matching that operates within the lower-dimensional latent feature space.
  arXiv  Detail & Related papers  (2023-11-03T04:29:49Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• In-Context Operator Learning with Data Prompts for Differential Equation Problems [12.61842281581773]
  This paper introduces a new neural-network-based approach, namely In-Context Operator Networks (ICON) ICON simultaneously learn operators from the prompted data and apply it to new questions during the inference stage, without any weight update. Our numerical results show the neural network's capability as a few-shot operator learner for a diversified type of differential equation problems.
  arXiv  Detail & Related papers  (2023-04-17T05:22:26Z)
• A PINN Approach to Symbolic Differential Operator Discovery with Sparse Data [0.0]
  In this work we perform symbolic discovery of differential operators in a situation where there is sparse experimental data. We modify the PINN approach by adding a neural network that learns a representation of unknown hidden terms in the differential equation. The algorithm yields both a surrogate solution to the differential equation and a black-box representation of the hidden terms.
  arXiv  Detail & Related papers  (2022-12-09T02:09:37Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)
• Liquid Time-constant Networks [117.57116214802504]
  We introduce a new class of time-continuous recurrent neural network models. Instead of declaring a learning system's dynamics by implicit nonlinearities, we construct networks of linear first-order dynamical systems. These neural networks exhibit stable and bounded behavior, yield superior expressivity within the family of neural ordinary differential equations.
  arXiv  Detail & Related papers  (2020-06-08T09:53:35Z)
• Hamiltonian neural networks for solving equations of motion [3.1498833540989413]
  We present a Hamiltonian neural network that solves the differential equations that govern dynamical systems. A symplectic Euler integrator requires two orders more evaluation points than the Hamiltonian network in order to achieve the same order of the numerical error.
  arXiv  Detail & Related papers  (2020-01-29T21:48:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.