LeMON: Learning to Learn Multi-Operator Networks

• URL: http://arxiv.org/abs/2408.16168v1
• Date: Wed, 28 Aug 2024 23:20:03 GMT
• Title: LeMON: Learning to Learn Multi-Operator Networks
• Authors: Jingmin Sun, Zecheng Zhang, Hayden Schaeffer,
• Abstract summary: Single-operator learning involves training a deep neural network to learn a specific operator. Recent work in multi-operator learning uses an operator embedding structure to train a single neural network on data from multiple operators. We propose pretraining and fine-tuning strategies for solving PDEs using multi-operator learning.
• Score: 0.6554326244334868
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Single-operator learning involves training a deep neural network to learn a specific operator, whereas recent work in multi-operator learning uses an operator embedding structure to train a single neural network on data from multiple operators. Thus, multi-operator learning is capable of predicting a range of operators within one model. In this work, we propose pretraining and fine-tuning strategies for solving PDEs using multi-operator learning. One key aspect is that by increasing the number of families of operators used in pretraining, a PDE foundation model can be fine-tuned to downstream tasks involving new PDEs with a limited number of samples, thus outperforming single operator neural networks. Specifically, a multi-operator learning model pre-trained with data from diverse PDE families can predict unseen operators after fine-tuning with only a limited number of operators from the new family, enabling them to serve as a data-free PDE solver. We also show that the proposed training and fine-tuning method is able to predict new operators in zero-shot prediction without samples. Additionally, we introduce a PDE-agnostic meta-learning algorithm to improve the adaptability of the model to various PDEs by providing a better parameter initialization process. To address the needs of applications with limited computing resources, we explore low-rank adaptation methods that reduce computational costs while enhancing solver accuracy. Lastly, by examining the scaling law with respect to the number of operator families, we establish and highlight its potential for broad adaptation in PDE-solving tasks.

Related papers

• Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation [69.60321475454843]
  We propose DPCPL, the first pre-training and prompt-tuning paradigm tailored for Multi-Behavior Sequential Recommendation. In the pre-training stage, we propose a novel Efficient Behavior Miner (EBM) to filter out the noise at multiple time scales. Subsequently, we propose to tune the pre-trained model in a highly efficient manner with the proposed Customized Prompt Learning (CPL) module.
  arXiv  Detail & Related papers  (2024-08-21T06:48:38Z)
• DeltaPhi: Learning Physical Trajectory Residual for PDE Solving [54.13671100638092]
  We propose and formulate the Physical Trajectory Residual Learning (DeltaPhi) We learn the surrogate model for the residual operator mapping based on existing neural operator networks. We conclude that, compared to direct learning, physical residual learning is preferred for PDE solving.
  arXiv  Detail & Related papers  (2024-06-14T07:45:07Z)
• MODNO: Multi Operator Learning With Distributed Neural Operators [0.8702432681310401]
  The study of operator learning involves the utilization of neural networks to approximate operators. Recent advances have led to the approximation of multiple operators using foundation models equipped with millions or billions of trainable parameters. We present a novel distributed training approach aimed at enabling a single neural operator with significantly fewer parameters to tackle multi-operator learning challenges.
  arXiv  Detail & Related papers  (2024-04-03T17:49:41Z)
• Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning [45.78096783448304]
  In this work, seeking data efficiency, we design unsupervised pretraining for PDE operator learning. We mine unlabeled PDE data without simulated solutions, and we pretrain neural operators with physics-inspired reconstruction-based proxy tasks. Our method is highly data-efficient, more generalizable, and even outperforms conventional vision-pretrained models.
  arXiv  Detail & Related papers  (2024-02-24T06:27:33Z)
• A foundational neural operator that continuously learns without forgetting [1.0878040851638]
  We introduce the concept of the Neural Combinatorial Wavelet Neural Operator (NCWNO) as a foundational model for scientific computing. The NCWNO is specifically designed to excel in learning from a diverse spectrum of physics and continuously adapt to the solution operators associated with parametric partial differential equations (PDEs) The proposed foundational model offers two key advantages: (i) it can simultaneously learn solution operators for multiple parametric PDEs, and (ii) it can swiftly generalize to new parametric PDEs with minimal fine-tuning.
  arXiv  Detail & Related papers  (2023-10-29T03:20:10Z)
• PILOT: A Pre-Trained Model-Based Continual Learning Toolbox [71.63186089279218]
  This paper introduces a pre-trained model-based continual learning toolbox known as PILOT. On the one hand, PILOT implements some state-of-the-art class-incremental learning algorithms based on pre-trained models, such as L2P, DualPrompt, and CODA-Prompt. On the other hand, PILOT fits typical class-incremental learning algorithms within the context of pre-trained models to evaluate their effectiveness.
  arXiv  Detail & Related papers  (2023-09-13T17:55:11Z)
• Variational operator learning: A unified paradigm marrying training neural operators and solving partial differential equations [9.148052787201797]
  We propose a novel paradigm that provides a unified framework of training neural operators and solving PDEs with the variational form. With a label-free training set and a 5-label-only shift set, VOL learns solution operators with its test errors decreasing in a power law with respect to the amount of unlabeled data.
  arXiv  Detail & Related papers  (2023-04-09T13:20:19Z)
• Reliable extrapolation of deep neural operators informed by physics or sparse observations [2.887258133992338]
  Deep neural operators can learn nonlinear mappings between infinite-dimensional function spaces via deep neural networks. DeepONets provide a new simulation paradigm in science and engineering. We propose five reliable learning methods that guarantee a safe prediction under extrapolation.
  arXiv  Detail & Related papers  (2022-12-13T03:02:46Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• Large-scale Neural Solvers for Partial Differential Equations [48.7576911714538]
  Solving partial differential equations (PDE) is an indispensable part of many branches of science as many processes can be modelled in terms of PDEs. Recent numerical solvers require manual discretization of the underlying equation as well as sophisticated, tailored code for distributed computing. We examine the applicability of continuous, mesh-free neural solvers for partial differential equations, physics-informed neural networks (PINNs) We discuss the accuracy of GatedPINN with respect to analytical solutions -- as well as state-of-the-art numerical solvers, such as spectral solvers.
  arXiv  Detail & Related papers  (2020-09-08T13:26:51Z)
• Self-Organized Operational Neural Networks with Generative Neurons [87.32169414230822]
  ONNs are heterogenous networks with a generalized neuron model that can encapsulate any set of non-linear operators. We propose Self-organized ONNs (Self-ONNs) with generative neurons that have the ability to adapt (optimize) the nodal operator of each connection.
  arXiv  Detail & Related papers  (2020-04-24T14:37:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.