Confidence estimation of classification based on the distribution of the neural network output layer

• URL: http://arxiv.org/abs/2210.07745v2
• Date: Tue, 18 Oct 2022 07:20:28 GMT
• Title: Confidence estimation of classification based on the distribution of the neural network output layer
• Authors: Abdel Aziz Taha, Leonhard Hennig, Petr Knoth
• Abstract summary: One of the most common problems preventing the application of prediction models in the real world is lack of generalization. We propose novel methods that estimate uncertainty of particular predictions generated by a neural network classification model. The proposed methods infer the confidence of a particular prediction based on the distribution of the logit values corresponding to this prediction.
• Score: 4.529188601556233
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: One of the most common problems preventing the application of prediction models in the real world is lack of generalization: The accuracy of models, measured in the benchmark does repeat itself on future data, e.g. in the settings of real business. There is relatively little methods exist that estimate the confidence of prediction models. In this paper, we propose novel methods that, given a neural network classification model, estimate uncertainty of particular predictions generated by this model. Furthermore, we propose a method that, given a model and a confidence level, calculates a threshold that separates prediction generated by this model into two subsets, one of them meets the given confidence level. In contrast to other methods, the proposed methods do not require any changes on existing neural networks, because they simply build on the output logit layer of a common neural network. In particular, the methods infer the confidence of a particular prediction based on the distribution of the logit values corresponding to this prediction. The proposed methods constitute a tool that is recommended for filtering predictions in the process of knowledge extraction, e.g. based on web scrapping, where predictions subsets are identified that maximize the precision on cost of the recall, which is less important due to the availability of data. The method has been tested on different tasks including relation extraction, named entity recognition and image classification to show the significant increase of accuracy achieved.

Related papers

• Tractable Function-Space Variational Inference in Bayesian Neural Networks [72.97620734290139]
  A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters. We propose a scalable function-space variational inference method that allows incorporating prior information. We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
  arXiv  Detail & Related papers  (2023-12-28T18:33:26Z)
• BayesNetCNN: incorporating uncertainty in neural networks for image-based classification tasks [0.29005223064604074]
  We propose a method to convert a standard neural network into a Bayesian neural network. We estimate the variability of predictions by sampling different networks similar to the original one at each forward pass. We test our model in a large cohort of brain images from Alzheimer's Disease patients.
  arXiv  Detail & Related papers  (2022-09-27T01:07:19Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration [2.8935588665357077]
  We propose an efficient yet general modelling approach for obtaining well-calibrated, trustworthy probabilities for samples obtained after a domain shift. We introduce a new training strategy combining an entropy-encouraging loss term with an adversarial calibration loss term and demonstrate that this results in well-calibrated and technically trustworthy predictions.
  arXiv  Detail & Related papers  (2020-12-20T13:39:29Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening [0.0]
  We investigate the effects of model architectures, regularization methods, and loss functions on the prediction performance and reliability of classification results. Our result highlights that correct choice of regularization and inference methods is evidently important to achieve high success rate.
  arXiv  Detail & Related papers  (2020-03-17T10:13:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.