UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation

• URL: http://arxiv.org/abs/2403.07187v4
• Date: Sat, 23 Nov 2024 16:39:43 GMT
• Title: UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
• Authors: Junhong Shen, Tanya Marwah, Ameet Talwalkar,
• Abstract summary: UPS embeds different PDEs into a shared representation space and processes them using a F-transformer architecture. Cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data transfer and 26 times less compute.
• Score: 25.063470461409686
• License:
• Abstract: We present Unified PDE Solvers (UPS), a data- and compute-efficient approach to developing unified neural operators for diverse families of spatiotemporal PDEs from various domains, dimensions, and resolutions. UPS embeds different PDEs into a shared representation space and processes them using a FNO-transformer architecture. Rather than training the network from scratch, which is data-demanding and computationally expensive, we warm-start the transformer from pretrained LLMs and perform explicit alignment to reduce the modality gap while improving data and compute efficiency. The cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data and 26 times less compute. Meanwhile, it is capable of few-shot transfer to unseen PDE families and coefficients.

Related papers

• Latent Neural Operator Pretraining for Solving Time-Dependent PDEs [5.8039987932401225]
  We propose the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone. Our proposed LNOP framework reduces the solution error by 31.7% on four problems and can be further improved to 57.1% after finetuning. These results show that our method is more competitive in terms of solution precision, transfer capability and data efficiency compared to non-pretrained neural operators.
  arXiv  Detail & Related papers  (2024-10-26T06:57:22Z)
• Unisolver: PDE-Conditional Transformers Are Universal PDE Solvers [55.0876373185983]
  We present the Universal PDE solver (Unisolver) capable of solving a wide scope of PDEs. Our key finding is that a PDE solution is fundamentally under the control of a series of PDE components. Unisolver achieves consistent state-of-the-art results on three challenging large-scale benchmarks.
  arXiv  Detail & Related papers  (2024-05-27T15:34:35Z)
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training [87.90342423839876]
  We present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories.
  arXiv  Detail & Related papers  (2024-03-06T08:38:34Z)
• Efficient Neural PDE-Solvers using Quantization Aware Training [71.0934372968972]
  We show that quantization can successfully lower the computational cost of inference while maintaining performance. Our results on four standard PDE datasets and three network architectures show that quantization-aware training works across settings and three orders of FLOPs magnitudes.
  arXiv  Detail & Related papers  (2023-08-14T09:21:19Z)
• Training Deep Surrogate Models with Large Scale Online Learning [48.7576911714538]
  Deep learning algorithms have emerged as a viable alternative for obtaining fast solutions for PDEs. Models are usually trained on synthetic data generated by solvers, stored on disk and read back for training. It proposes an open source online training framework for deep surrogate models.
  arXiv  Detail & Related papers  (2023-06-28T12:02:27Z)
• Neural Partial Differential Equations with Functional Convolution [30.35306295442881]
  We present a lightweighted neural PDE representation to discover the hidden structure and predict the solution of different nonlinear PDEs. We leverage the prior of translational similarity'' of numerical PDE differential operators to drastically reduce the scale of learning model and training data.
  arXiv  Detail & Related papers  (2023-03-10T04:25:38Z)
• MAgNet: Mesh Agnostic Neural PDE Solver [68.8204255655161]
  Climate predictions require fine-temporal resolutions to resolve all turbulent scales in the fluid simulations. Current numerical model solveers PDEs on grids that are too coarse (3km to 200km on each side) We design a novel architecture that predicts the spatially continuous solution of a PDE given a spatial position query.
  arXiv  Detail & Related papers  (2022-10-11T14:52:20Z)
• Lie Point Symmetry Data Augmentation for Neural PDE Solvers [69.72427135610106]
  We present a method, which can partially alleviate this problem, by improving neural PDE solver sample complexity. In the context of PDEs, it turns out that we are able to quantitatively derive an exhaustive list of data transformations. We show how it can easily be deployed to improve neural PDE solver sample complexity by an order of magnitude.
  arXiv  Detail & Related papers  (2022-02-15T18:43:17Z)
• Adversarial Multi-task Learning Enhanced Physics-informed Neural Networks for Solving Partial Differential Equations [9.823102211212582]
  We introduce the novel approach of employing multi-task learning techniques, the uncertainty-weighting loss and the gradients surgery, in the context of learning PDE solutions. In the experiments, our proposed methods are found to be effective and reduce the error on the unseen data points as compared to the previous approaches.
  arXiv  Detail & Related papers  (2021-04-29T13:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.