Explicit Regularisation in Gaussian Noise Injections

• URL: http://arxiv.org/abs/2007.07368v6
• Date: Tue, 19 Jan 2021 16:41:43 GMT
• Title: Explicit Regularisation in Gaussian Noise Injections
• Authors: Alexander Camuto, Matthew Willetts, Umut \c{S}im\c{s}ekli, Stephen Roberts, Chris Holmes
• Abstract summary: We study the regularisation induced in neural networks by Gaussian noise injections (GNIs) We derive the explicit regulariser of GNIs, obtained by marginalising out the injected noise. We show analytically and empirically that such regularisation produces calibrated classifiers with large classification margins.
• Score: 64.11680298737963
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the regularisation induced in neural networks by Gaussian noise injections (GNIs). Though such injections have been extensively studied when applied to data, there have been few studies on understanding the regularising effect they induce when applied to network activations. Here we derive the explicit regulariser of GNIs, obtained by marginalising out the injected noise, and show that it penalises functions with high-frequency components in the Fourier domain; particularly in layers closer to a neural network's output. We show analytically and empirically that such regularisation produces calibrated classifiers with large classification margins.

Related papers

• Sparse Variational Contaminated Noise Gaussian Process Regression with Applications in Geomagnetic Perturbations Forecasting [4.675221539472143]
  We propose a scalable inference algorithm for fitting sparse Gaussian process regression models with contaminated normal noise on large datasets. We show that our approach yields shorter prediction intervals for similar coverage and accuracy when compared to an artificial dense neural network baseline.
  arXiv  Detail & Related papers  (2024-02-27T15:08:57Z)
• Feature Noise Boosts DNN Generalization under Label Noise [65.36889005555669]
  The presence of label noise in the training data has a profound impact on the generalization of deep neural networks (DNNs) In this study, we introduce and theoretically demonstrate a simple feature noise method, which directly adds noise to the features of training data.
  arXiv  Detail & Related papers  (2023-08-03T08:31:31Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Explicit Regularization in Overparametrized Models via Noise Injection [14.492434617004932]
  We show that small perturbations induce explicit regularization for simple finite-dimensional models. We empirically show that the small perturbations lead to better generalization performance than vanilla (stochastic) gradient descent training.
  arXiv  Detail & Related papers  (2022-06-09T17:00:23Z)
• On the Inherent Regularization Effects of Noise Injection During Training [12.614901374282868]
  We present a theoretical study of one particular way of random perturbation, which corresponds to injecting artificial noise to the training data. We provide a precise characterization of the training and generalization errors of such randomly perturbed learning problems on a random feature model.
  arXiv  Detail & Related papers  (2021-02-15T07:43:18Z)
• Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections [73.95786440318369]
  We focus on the so-called implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of gradient descent (SGD) We show that this effect induces an asymmetric heavy-tailed noise on gradient updates. We then formally prove that GNIs induce an implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry.
  arXiv  Detail & Related papers  (2021-02-13T21:28:09Z)
• Gravitational-wave selection effects using neural-network classifiers [0.0]
  We train a series of neural-network classifiers to predict the LIGO/Virgo detectability of gravitational-wave signals from compact-binary mergers. We include the effect of spin precession, higher-order modes, and multiple detectors. Our approach is ready to be used in conjunction with full pipeline injections, thus paving the way toward including actual distributions of astrophysical and noise triggers into gravitational-wave population analyses.
  arXiv  Detail & Related papers  (2020-07-13T18:00:04Z)
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance [76.54300276636982]
  Noise in gradient descent provides a crucial implicit regularization effect for training over parameterized models. We show that parameter-dependent noise -- induced by mini-batches or label perturbation -- is far more effective than Gaussian noise. Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not.
  arXiv  Detail & Related papers  (2020-06-15T18:31:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.