Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty

• URL: http://arxiv.org/abs/2006.06399v3
• Date: Sat, 6 Feb 2021 08:27:27 GMT
• Title: Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Authors: Taejong Joo, Uijung Chung
• Abstract summary: In this work, we revisit the importance of explicit regularization for obtaining well-calibrated predictive uncertainty. We introduce a measure of calibration performance, which is lower bounded by the log-likelihood. We then explore explicit regularization techniques for improving the log-likelihood on unseen samples, which provides well-calibrated predictive uncertainty.
• Score: 6.09170287691728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: From the statistical learning perspective, complexity control via explicit regularization is a necessity for improving the generalization of over-parameterized models. However, the impressive generalization performance of neural networks with only implicit regularization may be at odds with this conventional wisdom. In this work, we revisit the importance of explicit regularization for obtaining well-calibrated predictive uncertainty. Specifically, we introduce a probabilistic measure of calibration performance, which is lower bounded by the log-likelihood. We then explore explicit regularization techniques for improving the log-likelihood on unseen samples, which provides well-calibrated predictive uncertainty. Our findings present a new direction to improve the predictive probability quality of deterministic neural networks, which can be an efficient and scalable alternative to Bayesian neural networks and ensemble methods.

Related papers

• On Measuring Calibration of Discrete Probabilistic Neural Networks [3.120856767382004]
  Training neural networks to fit high-dimensional probability distributions via maximum likelihood has become an effective method for uncertainty quantification. Traditional metrics like Expected Error (ECE) and Negative Log Likelihood (NLL) have limitations. This paper proposes a new approach using conditional kernel mean embeddings to measure calibration discrepancies without these biases and assumptions.
  arXiv  Detail & Related papers  (2024-05-20T23:30:07Z)
• Tractable Function-Space Variational Inference in Bayesian Neural Networks [72.97620734290139]
  A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters. We propose a scalable function-space variational inference method that allows incorporating prior information. We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
  arXiv  Detail & Related papers  (2023-12-28T18:33:26Z)
• Function-Space Regularization in Neural Networks: A Probabilistic Perspective [51.133793272222874]
  We show that we can derive a well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training. We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection and highly-calibrated predictive uncertainty estimates.
  arXiv  Detail & Related papers  (2023-12-28T17:50:56Z)
• Learning Expressive Priors for Generalization and Uncertainty Estimation in Neural Networks [77.89179552509887]
  We propose a novel prior learning method for advancing generalization and uncertainty estimation in deep neural networks. The key idea is to exploit scalable and structured posteriors of neural networks as informative priors with generalization guarantees. We exhaustively show the effectiveness of this method for uncertainty estimation and generalization.
  arXiv  Detail & Related papers  (2023-07-15T09:24:33Z)
• A Large-Scale Study of Probabilistic Calibration in Neural Network Regression [3.13468877208035]
  We conduct the largest empirical study to date to assess the probabilistic calibration of neural networks. We introduce novel differentiable recalibration and regularization methods, uncovering new insights into their effectiveness.
  arXiv  Detail & Related papers  (2023-06-05T09:33:39Z)
• Improved uncertainty quantification for neural networks with Bayesian last layer [0.0]
  Uncertainty quantification is an important task in machine learning. We present a reformulation of the log-marginal likelihood of a NN with BLL which allows for efficient training using backpropagation.
  arXiv  Detail & Related papers  (2023-02-21T20:23:56Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• Regularizing Class-wise Predictions via Self-knowledge Distillation [80.76254453115766]
  We propose a new regularization method that penalizes the predictive distribution between similar samples. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve the generalization ability.
  arXiv  Detail & Related papers  (2020-03-31T06:03:51Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.