Deep connections between learning from limited labels & physical parameter estimation -- inspiration for regularization

• URL: http://arxiv.org/abs/2003.07908v1
• Date: Tue, 17 Mar 2020 19:33:50 GMT
• Title: Deep connections between learning from limited labels & physical parameter estimation -- inspiration for regularization
• Authors: Bas Peters
• Abstract summary: We show that explicit regularization of model parameters in PDE constrained optimization translates to regularization of the network output. A hyperspectral imaging example shows that minimum prior information together with cross-validation for optimal regularization parameters boosts the segmentation accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently established equivalences between differential equations and the structure of neural networks enabled some interpretation of training of a neural network as partial-differential-equation (PDE) constrained optimization. We add to the previously established connections, explicit regularization that is particularly beneficial in the case of single large-scale examples with partial annotation. We show that explicit regularization of model parameters in PDE constrained optimization translates to regularization of the network output. Examination of the structure of the corresponding Lagrangian and backpropagation algorithm do not reveal additional computational challenges. A hyperspectral imaging example shows that minimum prior information together with cross-validation for optimal regularization parameters boosts the segmentation accuracy.

Related papers

• Neural Parameter Regression for Explicit Representations of PDE Solution Operators [22.355460388065964]
  We introduce Neural Regression (NPR), a novel framework specifically developed for learning solution operators in Partial Differential Equations (PDEs) NPR employs Physics-Informed Neural Network (PINN, Raissi et al., 2021) techniques to regress Neural Network (NN) parameters. The framework shows remarkable adaptability to new initial and boundary conditions, allowing for rapid fine-tuning and inference.
  arXiv  Detail & Related papers  (2024-03-19T14:30:56Z)
• Function-Space Regularization in Neural Networks: A Probabilistic Perspective [51.133793272222874]
  We show that we can derive a well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training. We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection and highly-calibrated predictive uncertainty estimates.
  arXiv  Detail & Related papers  (2023-12-28T17:50:56Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• ReLU Characteristic Activation Analysis [2.2713084727838115]
  We introduce a novel approach for analyzing the training dynamics of ReLU networks by examining the characteristic activation boundaries of individual neurons. Our proposed analysis reveals a critical instability in common neural network parameterizations and normalizations during convergence optimization, which impedes fast convergence and hurts performance.
  arXiv  Detail & Related papers  (2023-05-25T10:19:13Z)
• Learning Regularization Parameters of Inverse Problems via Deep Neural Networks [0.0]
  We consider a supervised learning approach, where a network is trained to approximate the mapping from observation data to regularization parameters. We show that a wide variety of regularization functionals, forward models, and noise models may be considered. The network-obtained regularization parameters can be computed more efficiently and may even lead to more accurate solutions.
  arXiv  Detail & Related papers  (2021-04-14T02:38:38Z)
• A Deep Learning approach to Reduced Order Modelling of Parameter Dependent Partial Differential Equations [0.2148535041822524]
  We develop a constructive approach based on Deep Neural Networks for the efficient approximation of the parameter-to-solution map. In particular, we consider parametrized advection-diffusion PDEs, and we test the methodology in the presence of strong transport fields.
  arXiv  Detail & Related papers  (2021-03-10T17:01:42Z)
• Joint Network Topology Inference via Structured Fusion Regularization [70.30364652829164]
  Joint network topology inference represents a canonical problem of learning multiple graph Laplacian matrices from heterogeneous graph signals. We propose a general graph estimator based on a novel structured fusion regularization. We show that the proposed graph estimator enjoys both high computational efficiency and rigorous theoretical guarantee.
  arXiv  Detail & Related papers  (2021-03-05T04:42:32Z)
• Iterative Surrogate Model Optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networks [14.380314061763508]
  We present a novel active learning algorithm, termed as iterative surrogate model optimization (ISMO) This algorithm is based on deep neural networks and its key feature is the iterative selection of training data through a feedback loop between deep neural networks and any underlying standard optimization algorithm.
  arXiv  Detail & Related papers  (2020-08-13T07:31:07Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Convex Geometry and Duality of Over-parameterized Neural Networks [70.15611146583068]
  We develop a convex analytic approach to analyze finite width two-layer ReLU networks. We show that an optimal solution to the regularized training problem can be characterized as extreme points of a convex set. In higher dimensions, we show that the training problem can be cast as a finite dimensional convex problem with infinitely many constraints.
  arXiv  Detail & Related papers  (2020-02-25T23:05:33Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.