Meta-learning PINN loss functions

• URL: http://arxiv.org/abs/2107.05544v1
• Date: Mon, 12 Jul 2021 16:13:39 GMT
• Title: Meta-learning PINN loss functions
• Authors: Apostolos F Psaros, Kenji Kawaguchi, George Em Karniadakis
• Abstract summary: We propose a meta-learning technique for offline discovery of physics-informed neural network (PINN) loss functions. We develop a gradient-based meta-learning algorithm for addressing diverse task distributions based on parametrized partial differential equations (PDEs) Our results indicate that significant performance improvement can be achieved by using a shared-among-tasks offline-learned loss function.
• Score: 5.543220407902113
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a meta-learning technique for offline discovery of physics-informed neural network (PINN) loss functions. We extend earlier works on meta-learning, and develop a gradient-based meta-learning algorithm for addressing diverse task distributions based on parametrized partial differential equations (PDEs) that are solved with PINNs. Furthermore, based on new theory we identify two desirable properties of meta-learned losses in PINN problems, which we enforce by proposing a new regularization method or using a specific parametrization of the loss function. In the computational examples, the meta-learned losses are employed at test time for addressing regression and PDE task distributions. Our results indicate that significant performance improvement can be achieved by using a shared-among-tasks offline-learned loss function even for out-of-distribution meta-testing. In this case, we solve for test tasks that do not belong to the task distribution used in meta-training, and we also employ PINN architectures that are different from the PINN architecture used in meta-training. To better understand the capabilities and limitations of the proposed method, we consider various parametrizations of the loss function and describe different algorithm design options and how they may affect meta-learning performance.

Related papers

• Fast and Efficient Local Search for Genetic Programming Based Loss Function Learning [12.581217671500887]
  We propose a new meta-learning framework for task and model-agnostic loss function learning via a hybrid search approach. Results show that the learned loss functions bring improved convergence, sample efficiency, and inference performance on tabulated, computer vision, and natural language processing problems.
  arXiv  Detail & Related papers  (2024-03-01T02:20:04Z)
• A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloading [62.34538208323411]
  We propose a multi-head ensemble multi-task learning (MEMTL) approach with a shared backbone and multiple prediction heads (PHs) MEMTL outperforms benchmark methods in both the inference accuracy and mean square error without requiring additional training data.
  arXiv  Detail & Related papers  (2023-09-02T11:01:16Z)
• A practical PINN framework for multi-scale problems with multi-magnitude loss terms [3.8645424244172135]
  We propose a practical deep learning framework for multi-scale problems using PINNs. New PINN methods differ from the conventional PINN method mainly in two aspects. The proposed methods significantly outperform the conventional PINN method in terms of computational efficiency and computational accuracy.
  arXiv  Detail & Related papers  (2023-08-13T03:26:01Z)
• Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning [70.52689048213398]
  This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features. We find new and interesting properties that do not exist in single-task linear regression. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
  arXiv  Detail & Related papers  (2023-04-09T20:36:13Z)
• Learning Symbolic Model-Agnostic Loss Functions via Meta-Learning [12.581217671500887]
  We propose a new meta-learning framework for learning model-agnostic loss functions via a hybrid neuro-symbolic search approach. Results show that the meta-learned loss functions discovered by the newly proposed method outperform both the cross-entropy loss and state-of-the-art loss function learning methods.
  arXiv  Detail & Related papers  (2022-09-19T10:29:01Z)
• InverseForm: A Loss Function for Structured Boundary-Aware Segmentation [80.39674800972182]
  We present a novel boundary-aware loss term for semantic segmentation using an inverse-transformation network. This plug-in loss term complements the cross-entropy loss in capturing boundary transformations. We analyze the quantitative and qualitative effects of our loss function on three indoor and outdoor segmentation benchmarks.
  arXiv  Detail & Related papers  (2021-04-06T18:52:45Z)
• Meta-Learning with Neural Tangent Kernels [58.06951624702086]
  We propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK) Within this paradigm, we introduce two meta-learning algorithms, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory.
  arXiv  Detail & Related papers  (2021-02-07T20:53:23Z)
• Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation [56.343646789922545]
  We propose to automate the design of metric-specific loss functions by searching differentiable surrogate losses for each metric. Experiments on PASCAL VOC and Cityscapes demonstrate that the searched surrogate losses outperform the manually designed loss functions consistently.
  arXiv  Detail & Related papers  (2020-10-15T17:59:08Z)
• Meta-Learning with Network Pruning [40.07436648243748]
  We propose a network pruning based meta-learning approach for overfitting reduction via explicitly controlling the capacity of network. We have implemented our approach on top of Reptile assembled with two network pruning routines: Dense-Sparse-Dense (DSD) and Iterative Hard Thresholding (IHT)
  arXiv  Detail & Related papers  (2020-07-07T06:13:11Z)
• MOPS-Net: A Matrix Optimization-driven Network forTask-Oriented 3D Point Cloud Downsampling [86.42733428762513]
  MOPS-Net is a novel interpretable deep learning-based method for matrix optimization. We show that MOPS-Net can achieve favorable performance against state-of-the-art deep learning-based methods over various tasks.
  arXiv  Detail & Related papers  (2020-05-01T14:01:53Z)
• Optimizing Loss Functions Through Multivariate Taylor Polynomial Parameterization [16.8615211682877]
  Loss functions are a type of metaknowledge that is crucial to effective training of deep neural network (DNN) architectures. This paper proposes continuous CMA-ES optimization of Taylor parameterizations. In MNIST, CIFAR-10, and SVHN benchmark tasks, TaylorGLO finds new loss functions that outperform functions previously discovered through GP.
  arXiv  Detail & Related papers  (2020-01-31T21:25:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.