Can Physics Informed Neural Operators Self Improve?

• URL: http://arxiv.org/abs/2311.13885v1
• Date: Thu, 23 Nov 2023 10:15:09 GMT
• Title: Can Physics Informed Neural Operators Self Improve?
• Abstract summary: We explore the use of self-training for learning fast solvers for systems of partial differential equations (Eg: Neural Operators) In this work we show that self-training can be used to close this gap in performance. Specifically, FNOs, when trained exclusively with physics loss through self-training, approach 1.07x for Burgers and 1.02x for Darcy, compared to FNOs trained with both data and physics loss.
• Score: 8.979310336973887
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Self-training techniques have shown remarkable value across many deep learning models and tasks. However, such techniques remain largely unexplored when considered in the context of learning fast solvers for systems of partial differential equations (Eg: Neural Operators). In this work, we explore the use of self-training for Fourier Neural Operators (FNO). Neural Operators emerged as a data driven technique, however, data from experiments or traditional solvers is not always readily available. Physics Informed Neural Operators (PINO) overcome this constraint by utilizing a physics loss for the training, however the accuracy of PINO trained without data does not match the performance obtained by training with data. In this work we show that self-training can be used to close this gap in performance. We examine canonical examples, namely the 1D-Burgers and 2D-Darcy PDEs, to showcase the efficacy of self-training. Specifically, FNOs, when trained exclusively with physics loss through self-training, approach 1.07x for Burgers and 1.02x for Darcy, compared to FNOs trained with both data and physics loss. Furthermore, we discover that pseudo-labels can be used for self-training without necessarily training to convergence in each iteration. A consequence of this is that we are able to discover self-training schedules that improve upon the baseline performance of PINO in terms of accuracy as well as time.

Related papers

• On the training of physics-informed neural operators for solving parametric partial differential equations [3.41772976335063]
  Physics-informed neural operators (PINOs) aim to learn solution operators for partial differential equations by using the governing physics as supervision.<n>We study key components of the PINO training pipeline, including architecture design, choice loss balancing, and collocation-point sampling strategy.<n>We find that several optimization pathologies previously identified in PINN training naturally arise in PINOs, including gradient conflicts and causal violation.
  arXiv  Detail & Related papers  (2026-06-04T13:36:22Z)
• Data-Efficient Neural Operator Training via Physics-Based Active Learning [0.0]
  Solving partial differential equations with neural operators significantly reduces computational costs but remains bottlenecked by high training data requirements.<n>We introduce physics-based acquisition - a novel active learning algorithm that leverages the partial differential equation residual to guide data selection.<n>We show that, in our experiments, physics-based acquisition consistently outperforms random acquisition and matches the state of the art in data efficiency.
  arXiv  Detail & Related papers  (2026-05-20T16:13:53Z)
• Translation Invariance of Neural Operators for the FitzHugh-Nagumo Model [0.0]
  Neural operators (NOs) are a powerful deep learning framework designed to learn the solution operator that arise from partial differential equations.<n>This study investigates NOs ability to capture stiff-temporal dynamics of the Fitz-Nagumo novel model, which describes excitable cells.<n>We benchmark seven NOs architectures and evaluate their performance on translated test dynamics.
  arXiv  Detail & Related papers  (2026-03-18T09:28:48Z)
• From Cheap Geometry to Expensive Physics: Elevating Neural Operators via Latent Shape Pretraining [4.838293051443775]
  Industrial design evaluation often relies on simulations of governing partial differential equations (PDEs)<n> Operator learning has emerged as a promising approach to accelerate PDE solution prediction.<n>We propose a two-stage framework to better exploit this abundant, physics-agnostic resource.
  arXiv  Detail & Related papers  (2025-09-30T05:01:31Z)
• DeepONet as a Multi-Operator Extrapolation Model: Distributed Pretraining with Physics-Informed Fine-Tuning [6.635683993472882]
  We propose a novel fine-tuning method to achieve multi-operator learning. Our approach combines distributed learning to integrate data from various operators in pre-training, while physics-informed methods enable zero-shot fine-tuning.
  arXiv  Detail & Related papers  (2024-11-11T18:58:46Z)
• Solving Oscillator Ordinary Differential Equations in the Time Domain with High Performance via Soft-constrained Physics-informed Neural Network with Small Data [0.6446246430600296]
  Physics-informed neural network (PINN) incorporates physical information and knowledge into network topology or computational processes as model priors.<n>This study aims to investigate the performance characteristics of the soft-constrained PINN method to solve typical linear and nonlinear ordinary differential equations.
  arXiv  Detail & Related papers  (2024-08-19T13:02:06Z)
• Neural Operators with Localized Integral and Differential Kernels [77.76991758980003]
  We present a principled approach to operator learning that can capture local features under two frameworks. We prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions.
  arXiv  Detail & Related papers  (2024-02-26T18:59:31Z)
• Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning [45.78096783448304]
  In this work, seeking data efficiency, we design unsupervised pretraining for PDE operator learning. We mine unlabeled PDE data without simulated solutions, and we pretrain neural operators with physics-inspired reconstruction-based proxy tasks. Our method is highly data-efficient, more generalizable, and even outperforms conventional vision-pretrained models.
  arXiv  Detail & Related papers  (2024-02-24T06:27:33Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere [53.63505583883769]
  We introduce Spherical FNOs (SFNOs) for learning operators on spherical geometries. SFNOs have important implications for machine learning-based simulation of climate dynamics.
  arXiv  Detail & Related papers  (2023-06-06T16:27:17Z)
• DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning [5.2319020651074215]
  We propose a Curricumum-guided Contrastive Learning framework for neural Predictor (DCLP) Our method simplifies the contrastive task by designing a novel curriculum to enhance the stability of unlabeled training data distribution. We experimentally demonstrate that DCLP has high accuracy and efficiency compared with existing predictors.
  arXiv  Detail & Related papers  (2023-02-25T08:16:21Z)
• Physics informed WNO [0.0]
  We propose a physics-informed Wavelet Operator (WNO) for learning the solution operators of families of parametric partial differential equations (PDEs) without labeled training data. The efficacy of the framework is validated and illustrated with four nonlinear neural systems relevant to various fields of engineering and science.
  arXiv  Detail & Related papers  (2023-02-12T14:31:50Z)
• Certified machine learning: A posteriori error estimation for physics-informed neural networks [0.0]
  PINNs are known to be robust for smaller training sets, derive better generalization problems, and are faster to train. We show that using PINNs in comparison with purely data-driven neural networks is not only favorable for training performance but allows us to extract significant information on the quality of the approximated solution.
  arXiv  Detail & Related papers  (2022-03-31T14:23:04Z)
• Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability [53.27240222619834]
  Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks. Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
  arXiv  Detail & Related papers  (2022-03-10T06:23:41Z)
• Deep Reinforcement Learning Assisted Federated Learning Algorithm for Data Management of IIoT [82.33080550378068]
  The continuous expanded scale of the industrial Internet of Things (IIoT) leads to IIoT equipments generating massive amounts of user data every moment. How to manage these time series data in an efficient and safe way in the field of IIoT is still an open issue. This paper studies the FL technology applications to manage IIoT equipment data in wireless network environments.
  arXiv  Detail & Related papers  (2022-02-03T07:12:36Z)
• How does unlabeled data improve generalization in self-training? A one-hidden-layer theoretical analysis [93.37576644429578]
  This work establishes the first theoretical analysis for the known iterative self-training paradigm. We prove the benefits of unlabeled data in both training convergence and generalization ability. Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training.
  arXiv  Detail & Related papers  (2022-01-21T02:16:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.