Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings

• URL: http://arxiv.org/abs/2107.11496v1
• Date: Sat, 24 Jul 2021 00:43:17 GMT
• Title: Training multi-objective/multi-task collocation physics-informed neural network with student/teachers transfer learnings
• Authors: Bahador Bahmani and WaiChing Sun
• Abstract summary: This paper presents a PINN training framework that employs pre-training steps and a net-to-net knowledge transfer algorithm. A multi-objective optimization algorithm may improve the performance of a physical-informed neural network with competing constraints.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a PINN training framework that employs (1) pre-training steps that accelerates and improve the robustness of the training of physics-informed neural network with auxiliary data stored in point clouds, (2) a net-to-net knowledge transfer algorithm that improves the weight initialization of the neural network and (3) a multi-objective optimization algorithm that may improve the performance of a physical-informed neural network with competing constraints. We consider the training and transfer and multi-task learning of physics-informed neural network (PINN) as multi-objective problems where the physics constraints such as the governing equation, boundary conditions, thermodynamic inequality, symmetry, and invariant properties, as well as point cloud used for pre-training can sometimes lead to conflicts and necessitating the seek of the Pareto optimal solution. In these situations, weighted norms commonly used to handle multiple constraints may lead to poor performance, while other multi-objective algorithms may scale poorly with increasing dimensionality. To overcome this technical barrier, we adopt the concept of vectorized objective function and modify a gradient descent approach to handle the issue of conflicting gradients. Numerical experiments are compared the benchmark boundary value problems solved via PINN. The performance of the proposed paradigm is compared against the classical equal-weighted norm approach. Our numerical experiments indicate that the brittleness and lack of robustness demonstrated in some PINN implementations can be overcome with the proposed strategy.

Related papers

• WANCO: Weak Adversarial Networks for Constrained Optimization problems [5.257895611010853]
  We first transform minimax problems into minimax problems using the augmented Lagrangian method. We then use two (or several) deep neural networks to represent the primal and dual variables respectively. The parameters in the neural networks are then trained by an adversarial process.
  arXiv  Detail & Related papers  (2024-07-04T05:37:48Z)
• Enriched Physics-informed Neural Networks for Dynamic Poisson-Nernst-Planck Systems [0.8192907805418583]
  This paper proposes a meshless deep learning algorithm, enriched physics-informed neural networks (EPINNs) to solve dynamic Poisson-Nernst-Planck (PNP) equations. The EPINNs takes the traditional physics-informed neural networks as the foundation framework, and adds the adaptive loss weight to balance the loss functions. Numerical results indicate that the new method has better applicability than traditional numerical methods in solving such coupled nonlinear systems.
  arXiv  Detail & Related papers  (2024-02-01T02:57:07Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• Training Physics-Informed Neural Networks via Multi-Task Optimization for Traffic Density Prediction [3.3823703740215865]
  Physics-informed neural networks (PINNs) are a newly emerging research frontier in machine learning. We propose a new PINN training framework based on the multi-task optimization (MTO) paradigm. We implement the proposed framework and apply it to train the PINN for addressing the traffic density prediction problem.
  arXiv  Detail & Related papers  (2023-07-08T07:11:52Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach [10.250994619846416]
  We present a new training paradigm referred to as "gradient boosting" (GB) Instead of learning the solution of a given PDE using a single neural network directly, our algorithm employs a sequence of neural networks to achieve a superior outcome. This work also unlocks the door to employing ensemble learning techniques in PINNs.
  arXiv  Detail & Related papers  (2023-02-25T19:11:44Z)
• NeuralStagger: Accelerating Physics-constrained Neural PDE Solver with Spatial-temporal Decomposition [67.46012350241969]
  This paper proposes a general acceleration methodology called NeuralStagger. It decomposing the original learning tasks into several coarser-resolution subtasks. We demonstrate the successful application of NeuralStagger on 2D and 3D fluid dynamics simulations.
  arXiv  Detail & Related papers  (2023-02-20T19:36:52Z)
• Physics and Equality Constrained Artificial Neural Networks: Application to Partial Differential Equations [1.370633147306388]
  Physics-informed neural networks (PINNs) have been proposed to learn the solution of partial differential equations (PDE) Here, we show that this specific way of formulating the objective function is the source of severe limitations in the PINN approach. We propose a versatile framework that can tackle both inverse and forward problems.
  arXiv  Detail & Related papers  (2021-09-30T05:55:35Z)
• Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks [93.4221402881609]
  We present an in-depth, fine-grained ablation study of the effect of different weights initialization on the final distributions of weights and activations of different CNN architectures. To our best knowledge, we are the first to perform such a low-level, in-depth quantitative analysis of weights initialization and its effect on quantized behaviour.
  arXiv  Detail & Related papers  (2020-11-30T06:54:28Z)
• Deep Multi-Task Learning for Cooperative NOMA: System Design and Principles [52.79089414630366]
  We develop a novel deep cooperative NOMA scheme, drawing upon the recent advances in deep learning (DL) We develop a novel hybrid-cascaded deep neural network (DNN) architecture such that the entire system can be optimized in a holistic manner.
  arXiv  Detail & Related papers  (2020-07-27T12:38:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.