Latent Neural Operator Pretraining for Solving Time-Dependent PDEs

• URL: http://arxiv.org/abs/2410.20100v1
• Date: Sat, 26 Oct 2024 06:57:22 GMT
• Title: Latent Neural Operator Pretraining for Solving Time-Dependent PDEs
• Authors: Tian Wang, Chuang Wang,
• Abstract summary: We propose the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone. Our proposed LNOP framework reduces the solution error by 31.7% on four problems and can be further improved to 57.1% after finetuning. These results show that our method is more competitive in terms of solution precision, transfer capability and data efficiency compared to non-pretrained neural operators.
• Score: 5.8039987932401225
• License:
• Abstract: Pretraining methods gain increasing attraction recently for solving PDEs with neural operators. It alleviates the data scarcity problem encountered by neural operator learning when solving single PDE via training on large-scale datasets consisting of various PDEs and utilizing shared patterns among different PDEs to improve the solution precision. In this work, we propose the Latent Neural Operator Pretraining (LNOP) framework based on the Latent Neural Operator (LNO) backbone. We achieve universal transformation through pretraining on hybrid time-dependent PDE dataset to extract representations of different physical systems and solve various time-dependent PDEs in the latent space through finetuning on single PDE dataset. Our proposed LNOP framework reduces the solution error by 31.7% on four problems and can be further improved to 57.1% after finetuning. On out-of-distribution dataset, our LNOP model achieves roughly 50% lower error and 3$\times$ data efficiency on average across different dataset sizes. These results show that our method is more competitive in terms of solution precision, transfer capability and data efficiency compared to non-pretrained neural operators.

Related papers

• Active Learning for Neural PDE Solvers [18.665448858377694]
  Active Learning could help surrogate models reach the same accuracy with smaller training sets. We introduce AL4PDE, a modular and active learning benchmark. We show that AL reduces the average error by up to 71% compared to random sampling.
  arXiv  Detail & Related papers  (2024-08-02T18:48:58Z)
• Pretraining a Neural Operator in Lower Dimensions [7.136205674624813]
  We aim to Pretrain neural PDE solvers on Lower Dimensional PDEs (PreLowD) where data collection is the least expensive. We evaluate the effectiveness of this pretraining strategy in similar PDEs in higher dimensions. Our work sheds light on the effect of the fine-tuning configuration to make the most of this pretraining strategy.
  arXiv  Detail & Related papers  (2024-07-24T20:06:12Z)
• UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation [25.063470461409686]
  UPS embeds different PDEs into a shared representation space and processes them using a F-transformer architecture. Cross-modal UPS achieves state-of-the-art results on a wide range of 1D and 2D PDE families from PDEBench, outperforming existing unified models using 4 times less data transfer and 26 times less compute.
  arXiv  Detail & Related papers  (2024-03-11T22:00:39Z)
• DPOT: Auto-Regressive Denoising Operator Transformer for Large-Scale PDE Pre-Training [87.90342423839876]
  We present a new auto-regressive denoising pre-training strategy, which allows for more stable and efficient pre-training on PDE data. We train our PDE foundation model with up to 0.5B parameters on 10+ PDE datasets with more than 100k trajectories.
  arXiv  Detail & Related papers  (2024-03-06T08:38:34Z)
• Deep Equilibrium Based Neural Operators for Steady-State PDEs [100.88355782126098]
  We study the benefits of weight-tied neural network architectures for steady-state PDEs. We propose FNO-DEQ, a deep equilibrium variant of the FNO architecture that directly solves for the solution of a steady-state PDE.
  arXiv  Detail & Related papers  (2023-11-30T22:34:57Z)
• Training Deep Surrogate Models with Large Scale Online Learning [48.7576911714538]
  Deep learning algorithms have emerged as a viable alternative for obtaining fast solutions for PDEs. Models are usually trained on synthetic data generated by solvers, stored on disk and read back for training. It proposes an open source online training framework for deep surrogate models.
  arXiv  Detail & Related papers  (2023-06-28T12:02:27Z)
• Neural Partial Differential Equations with Functional Convolution [30.35306295442881]
  We present a lightweighted neural PDE representation to discover the hidden structure and predict the solution of different nonlinear PDEs. We leverage the prior of translational similarity'' of numerical PDE differential operators to drastically reduce the scale of learning model and training data.
  arXiv  Detail & Related papers  (2023-03-10T04:25:38Z)
• DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization [51.517956081644186]
  We introduce a new graph-based diffusion framework, namely DIFUSCO. Our framework casts NPC problems as discrete 0, 1-vector optimization problems. For the MIS problem, DIFUSCO outperforms the previous state-of-the-art neural solver on the challenging SATLIB benchmark.
  arXiv  Detail & Related papers  (2023-02-16T11:13:36Z)
• Lie Point Symmetry Data Augmentation for Neural PDE Solvers [69.72427135610106]
  We present a method, which can partially alleviate this problem, by improving neural PDE solver sample complexity. In the context of PDEs, it turns out that we are able to quantitatively derive an exhaustive list of data transformations. We show how it can easily be deployed to improve neural PDE solver sample complexity by an order of magnitude.
  arXiv  Detail & Related papers  (2022-02-15T18:43:17Z)
• Physics-Informed Neural Operator for Learning Partial Differential Equations [55.406540167010014]
  PINO is the first hybrid approach incorporating data and PDE constraints at different resolutions to learn the operator. The resulting PINO model can accurately approximate the ground-truth solution operator for many popular PDE families.
  arXiv  Detail & Related papers  (2021-11-06T03:41:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.