Masked Autoencoders are PDE Learners

• URL: http://arxiv.org/abs/2403.17728v2
• Date: Wed, 29 May 2024 16:14:23 GMT
• Title: Masked Autoencoders are PDE Learners
• Authors: Anthony Zhou, Amir Barati Farimani,
• Abstract summary: Masked pretraining can consolidate heterogeneous physics to learn latent representations and perform latent PDE arithmetic. neural solvers on learned latent representations can improve time-stepping and super-resolution performance across a variety of coefficients, discretizations, or boundary conditions.
• Score: 7.136205674624813
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural solvers for partial differential equations (PDEs) have great potential to generate fast and accurate physics solutions, yet their practicality is currently limited by their generalizability. PDEs evolve over broad scales and exhibit diverse behaviors; predicting these phenomena will require learning representations across a wide variety of inputs which may encompass different coefficients, boundary conditions, resolutions, or even equations. As a step towards generalizable PDE modeling, we adapt masked pretraining for physics problems. Through self-supervised learning across PDEs, masked autoencoders can consolidate heterogeneous physics to learn meaningful latent representations and perform latent PDE arithmetic in this space. Furthermore, we demonstrate that masked pretraining can improve PDE coefficient regression and the classification of PDE features. Lastly, conditioning neural solvers on learned latent representations can improve time-stepping and super-resolution performance across a variety of coefficients, discretizations, or boundary conditions, as well as on unseen PDEs. We hope that masked pretraining can emerge as a unifying method across large, unlabeled, and heterogeneous datasets to learn latent physics at scale.

Related papers

• DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning [63.5925701087252]
  We introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets.
  arXiv  Detail & Related papers  (2024-10-08T10:48:50Z)
• Text2PDE: Latent Diffusion Models for Accessible Physics Simulation [7.16525545814044]
  We introduce several methods to apply latent diffusion models to physics simulation. We show that the proposed approach is competitive with current neural PDE solvers in both accuracy and efficiency. By introducing a scalable, accurate, and usable physics simulator, we hope to bring neural PDE solvers closer to practical use.
  arXiv  Detail & Related papers  (2024-10-02T01:09:47Z)
• Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers [55.0876373185983]
  We present the Universal PDE solver (Unisolver) capable of solving a wide scope of PDEs. Our key finding is that a PDE solution is fundamentally under the control of a series of PDE components. Unisolver achieves consistent state-of-the-art results on three challenging large-scale benchmarks.
  arXiv  Detail & Related papers  (2024-05-27T15:34:35Z)
• Physics-informed Discretization-independent Deep Compositional Operator Network [1.2430809884830318]
  We introduce a novel physics-informed model architecture which can generalize to various discrete representations of PDE parameters and irregular domain shapes. Inspired by deep operator neural networks, our model involves a discretization-independent learning of parameter embedding repeatedly. Numerical results demonstrate the accuracy and efficiency of the proposed method.
  arXiv  Detail & Related papers  (2024-04-21T12:41:30Z)
• Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs [85.40198664108624]
  We propose Codomain Attention Neural Operator (CoDA-NO) to solve multiphysics problems with PDEs. CoDA-NO tokenizes functions along the codomain or channel space, enabling self-supervised learning or pretraining of multiple PDE systems. We find CoDA-NO to outperform existing methods by over 36% on complex downstream tasks with limited data.
  arXiv  Detail & Related papers  (2024-03-19T08:56:20Z)
• Deep Equilibrium Based Neural Operators for Steady-State PDEs [100.88355782126098]
  We study the benefits of weight-tied neural network architectures for steady-state PDEs. We propose FNO-DEQ, a deep equilibrium variant of the FNO architecture that directly solves for the solution of a steady-state PDE.
  arXiv  Detail & Related papers  (2023-11-30T22:34:57Z)
• Reduced-order modeling for parameterized PDEs via implicit neural representations [4.135710717238787]
  We present a new data-driven reduced-order modeling approach to efficiently solve parametrized partial differential equations (PDEs) The proposed framework encodes PDE and utilizes a parametrized neural ODE (PNODE) to learn latent dynamics characterized by multiple PDE parameters. We evaluate the proposed method at a large Reynolds number and obtain up to speedup of O(103) and 1% relative error to the ground truth values.
  arXiv  Detail & Related papers  (2023-11-28T01:35:06Z)
• Solving High-Dimensional PDEs with Latent Spectral Models [74.1011309005488]
  We present Latent Spectral Models (LSM) toward an efficient and precise solver for high-dimensional PDEs. Inspired by classical spectral methods in numerical analysis, we design a neural spectral block to solve PDEs in the latent space. LSM achieves consistent state-of-the-art and yields a relative gain of 11.5% averaged on seven benchmarks.
  arXiv  Detail & Related papers  (2023-01-30T04:58:40Z)
• Learning to Accelerate Partial Differential Equations via Latent Global Evolution [64.72624347511498]
  Latent Evolution of PDEs (LE-PDE) is a simple, fast and scalable method to accelerate the simulation and inverse optimization of PDEs. We introduce new learning objectives to effectively learn such latent dynamics to ensure long-term stability. We demonstrate up to 128x reduction in the dimensions to update, and up to 15x improvement in speed, while achieving competitive accuracy.
  arXiv  Detail & Related papers  (2022-06-15T17:31:24Z)
• Neural-PDE: A RNN based neural network for solving time dependent PDEs [6.560798708375526]
  Partial differential equations (PDEs) play a crucial role in studying a vast number of problems in science and engineering. We propose a sequence deep learning framework called Neural-PDE, which allows to automatically learn governing rules of any time-dependent PDE system. In our experiments the Neural-PDE can efficiently extract the dynamics within 20 epochs training, and produces accurate predictions.
  arXiv  Detail & Related papers  (2020-09-08T15:46:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.