Scalable Transformer for PDE Surrogate Modeling

• URL: http://arxiv.org/abs/2305.17560v2
• Date: Fri, 3 Nov 2023 01:32:08 GMT
• Title: Scalable Transformer for PDE Surrogate Modeling
• Authors: Zijie Li, Dule Shu, Amir Barati Farimani
• Abstract summary: Transformer has emerged as a promising tool for surrogate modeling of partial differential equations (PDEs) We propose Factorized Transformer (FactFormer), which is based on an axial factorized kernel integral. We showcase that the proposed model is able to simulate 2D Kolmogorov flow on a $256times 256$ grid and 3D smoke buoyancy on a $64times64times64$ grid with good accuracy and efficiency.
• Score: 9.438207505148947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer has shown state-of-the-art performance on various applications and has recently emerged as a promising tool for surrogate modeling of partial differential equations (PDEs). Despite the introduction of linear-complexity attention, applying Transformer to problems with a large number of grid points can be numerically unstable and computationally expensive. In this work, we propose Factorized Transformer (FactFormer), which is based on an axial factorized kernel integral. Concretely, we introduce a learnable projection operator that decomposes the input function into multiple sub-functions with one-dimensional domain. These sub-functions are then evaluated and used to compute the instance-based kernel with an axial factorized scheme. We showcase that the proposed model is able to simulate 2D Kolmogorov flow on a $256\times 256$ grid and 3D smoke buoyancy on a $64\times64\times64$ grid with good accuracy and efficiency. The proposed factorized scheme can serve as a computationally efficient low-rank surrogate for the full attention scheme when dealing with multi-dimensional problems.

Related papers

• Neurons for Neutrons: A Transformer Model for Computation Load Estimation on Domain-Decomposed Neutron Transport Problems [48.35237609036802]
  We propose a Transformer model with a unique 3D input embedding, and input representations designed for domain-decomposed neutron transport problems. We demonstrate that such a model trained on domain-decomposed Small Modular Reactor (SMR) simulations achieves 98.2% accuracy while being able to skip the small-scale simulation step entirely.
  arXiv  Detail & Related papers  (2024-11-05T18:17:51Z)
• Deep Neural Implicit Representation of Accessibility for Multi-Axis Manufacturing [0.0]
  We develop an implicit representation of the collision measure field via deep neural networks (DNNs) We show that our approach is able to accurately interpolate the collision measure from a sparse sampling of rotations, and can represent the collision measure field with a small memory footprint.
  arXiv  Detail & Related papers  (2024-08-30T06:27:25Z)
• SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation [74.07836010698801]
  We propose an SMPL-based Transformer framework (SMPLer) to address this issue. SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation. Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods.
  arXiv  Detail & Related papers  (2024-04-23T17:59:59Z)
• HAMLET: Graph Transformer Neural Operator for Partial Differential Equations [13.970458554623939]
  We present a novel graph transformer framework, HAMLET, designed to address the challenges in solving partial differential equations (PDEs) using neural networks. The framework uses graph transformers with modular input encoders to directly incorporate differential equation information into the solution process. Notably, HAMLET scales effectively with increasing data complexity and noise, showcasing its robustness.
  arXiv  Detail & Related papers  (2024-02-05T21:55:24Z)
• Transolver: A Fast Transformer Solver for PDEs on General Geometries [66.82060415622871]
  We present Transolver, which learns intrinsic physical states hidden behind discretized geometries. By calculating attention to physics-aware tokens encoded from slices, Transovler can effectively capture intricate physical correlations. Transolver achieves consistent state-of-the-art with 22% relative gain across six standard benchmarks and also excels in large-scale industrial simulations.
  arXiv  Detail & Related papers  (2024-02-04T06:37:38Z)
• Geometry-Informed Neural Operator for Large-Scale 3D PDEs [76.06115572844882]
  We propose the geometry-informed neural operator (GINO) to learn the solution operator of large-scale partial differential equations. We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points.
  arXiv  Detail & Related papers  (2023-09-01T16:59:21Z)
• FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics [8.629912408966145]
  We revisit the Taylor series expansion from a modern Machine Learning perspective. We introduce the Fast Continuous Convolutional Taylor Transform (FC2T2), a variant of the Fast Multipole Method (FMM), that allows for the efficient approximation of low dimensional convolutional operators in continuous space.
  arXiv  Detail & Related papers  (2021-10-29T22:58:42Z)
• PnP-DETR: Towards Efficient Visual Analysis with Transformers [146.55679348493587]
  Recently, DETR pioneered the solution vision tasks with transformers, it directly translates the image feature map into the object result. Recent transformer-based image recognition model andTT show consistent efficiency gain.
  arXiv  Detail & Related papers  (2021-09-15T01:10:30Z)
• Data-Driven Shadowgraph Simulation of a 3D Object [50.591267188664666]
  We are replacing the numerical code by a computationally cheaper projection based surrogate model. The model is able to approximate the electric fields at a given time without computing all preceding electric fields as required by numerical methods. This model has shown a good quality reconstruction in a problem of perturbation of data within a narrow range of simulation parameters and can be used for input data of large size.
  arXiv  Detail & Related papers  (2021-06-01T08:46:04Z)
• Center Smoothing for Certifiably Robust Vector-Valued Functions [59.46976586742266]
  We produce certifiable robustness for vector-valued functions bound to change in output caused by a small change in input. We demonstrate the effectiveness of our method on multiple learning tasks involving vector-valued functions with a wide range of input and output dimensionalities.
  arXiv  Detail & Related papers  (2021-02-19T01:34:48Z)
• The Random Feature Model for Input-Output Maps between Banach Spaces [6.282068591820945]
  The random feature model is a parametric approximation to kernel or regression methods. We propose a methodology for use of the random feature model as a data-driven surrogate for operators that map an input Banach space to an output Banach space.
  arXiv  Detail & Related papers  (2020-05-20T17:41:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.