Scalable Transformer for PDE Surrogate Modeling
- URL: http://arxiv.org/abs/2305.17560v2
- Date: Fri, 3 Nov 2023 01:32:08 GMT
- Title: Scalable Transformer for PDE Surrogate Modeling
- Authors: Zijie Li, Dule Shu, Amir Barati Farimani
- Abstract summary: Transformer has emerged as a promising tool for surrogate modeling of partial differential equations (PDEs)
We propose Factorized Transformer (FactFormer), which is based on an axial factorized kernel integral.
We showcase that the proposed model is able to simulate 2D Kolmogorov flow on a $256times 256$ grid and 3D smoke buoyancy on a $64times64times64$ grid with good accuracy and efficiency.
- Score: 9.438207505148947
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Transformer has shown state-of-the-art performance on various applications
and has recently emerged as a promising tool for surrogate modeling of partial
differential equations (PDEs). Despite the introduction of linear-complexity
attention, applying Transformer to problems with a large number of grid points
can be numerically unstable and computationally expensive. In this work, we
propose Factorized Transformer (FactFormer), which is based on an axial
factorized kernel integral. Concretely, we introduce a learnable projection
operator that decomposes the input function into multiple sub-functions with
one-dimensional domain. These sub-functions are then evaluated and used to
compute the instance-based kernel with an axial factorized scheme. We showcase
that the proposed model is able to simulate 2D Kolmogorov flow on a $256\times
256$ grid and 3D smoke buoyancy on a $64\times64\times64$ grid with good
accuracy and efficiency. The proposed factorized scheme can serve as a
computationally efficient low-rank surrogate for the full attention scheme when
dealing with multi-dimensional problems.
Related papers
- SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation [74.07836010698801]
We propose an SMPL-based Transformer framework (SMPLer) to address this issue.
SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation.
Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods.
arXiv Detail & Related papers (2024-04-23T17:59:59Z) - HAMLET: Graph Transformer Neural Operator for Partial Differential
Equations [6.699756195061548]
We present a novel graph transformer framework, HAMLET, designed to address the challenges in solving partial differential equations (PDEs) using neural networks.
The framework uses graph transformers with modular input encoders to directly incorporate differential equation information into the solution process.
Notably, HAMLET scales effectively with increasing data complexity and noise, showcasing its robustness.
arXiv Detail & Related papers (2024-02-05T21:55:24Z) - Transolver: A Fast Transformer Solver for PDEs on General Geometries [66.82060415622871]
We present Transolver, which learns intrinsic physical states hidden behind discretized geometries.
By calculating attention to physics-aware tokens encoded from slices, Transovler can effectively capture intricate physical correlations.
Transolver achieves consistent state-of-the-art with 22% relative gain across six standard benchmarks and also excels in large-scale industrial simulations.
arXiv Detail & Related papers (2024-02-04T06:37:38Z) - Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt
Tensor Products [16.84090726181652]
We propose a systematic approach to accelerate the complexity of the tensor products of irreps.
We introduce the Gaunt Product, which serves as a new method to construct efficient equivariant operations.
Our experiments on the Open Catalyst Project and 3BPA datasets demonstrate both the increased efficiency and improved performance.
arXiv Detail & Related papers (2024-01-18T18:57:10Z) - Geometry-Informed Neural Operator for Large-Scale 3D PDEs [76.06115572844882]
We propose the geometry-informed neural operator (GINO) to learn the solution operator of large-scale partial differential equations.
We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points.
arXiv Detail & Related papers (2023-09-01T16:59:21Z) - Digital quantum simulation of Schr\"odinger dynamics using adaptive
approximations of potential functions [1.9723551683930771]
Digital quantum simulation (DQS) of continuous-variable quantum systems in the position basis requires efficient implementation of diagonal unitaries approxing the time evolution operator generated by the potential energy function.
We show how adaptive grids can significantly reduce the total gate count at the cost of introducing a small number of ancillary qubits.
We demonstrate the circuit construction with both physically motivated and artificially designed potential functions, and discuss their generalizations to higher dimensions.
arXiv Detail & Related papers (2022-12-09T15:51:27Z) - FC2T2: The Fast Continuous Convolutional Taylor Transform with
Applications in Vision and Graphics [8.629912408966145]
We revisit the Taylor series expansion from a modern Machine Learning perspective.
We introduce the Fast Continuous Convolutional Taylor Transform (FC2T2), a variant of the Fast Multipole Method (FMM), that allows for the efficient approximation of low dimensional convolutional operators in continuous space.
arXiv Detail & Related papers (2021-10-29T22:58:42Z) - PnP-DETR: Towards Efficient Visual Analysis with Transformers [146.55679348493587]
Recently, DETR pioneered the solution vision tasks with transformers, it directly translates the image feature map into the object result.
Recent transformer-based image recognition model andTT show consistent efficiency gain.
arXiv Detail & Related papers (2021-09-15T01:10:30Z) - Data-Driven Shadowgraph Simulation of a 3D Object [50.591267188664666]
We are replacing the numerical code by a computationally cheaper projection based surrogate model.
The model is able to approximate the electric fields at a given time without computing all preceding electric fields as required by numerical methods.
This model has shown a good quality reconstruction in a problem of perturbation of data within a narrow range of simulation parameters and can be used for input data of large size.
arXiv Detail & Related papers (2021-06-01T08:46:04Z) - Center Smoothing for Certifiably Robust Vector-Valued Functions [59.46976586742266]
We produce certifiable robustness for vector-valued functions bound to change in output caused by a small change in input.
We demonstrate the effectiveness of our method on multiple learning tasks involving vector-valued functions with a wide range of input and output dimensionalities.
arXiv Detail & Related papers (2021-02-19T01:34:48Z) - The Random Feature Model for Input-Output Maps between Banach Spaces [6.282068591820945]
The random feature model is a parametric approximation to kernel or regression methods.
We propose a methodology for use of the random feature model as a data-driven surrogate for operators that map an input Banach space to an output Banach space.
arXiv Detail & Related papers (2020-05-20T17:41:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.