PaRCE: Probabilistic and Reconstruction-based Competency Estimation for CNN-based Image Classification

• URL: http://arxiv.org/abs/2411.16715v1
• Date: Fri, 22 Nov 2024 22:08:57 GMT
• Title: PaRCE: Probabilistic and Reconstruction-based Competency Estimation for CNN-based Image Classification
• Authors: Sara Pohland, Claire Tomlin,
• Abstract summary: We develop a probabilistic and reconstruction-based competency estimation (PaRCE) method. We find that our method can best distinguish between correctly classified, misclassified, and OOD samples with anomalous regions. Our method generates interpretable scores that most reliably capture a holistic notion of perception model confidence.
• Score: 0.10923877073891446
• License:
• Abstract: Convolutional neural networks (CNNs) are extremely popular and effective for image classification tasks but tend to be overly confident in their predictions. Various works have sought to quantify uncertainty associated with these models, detect out-of-distribution (OOD) inputs, or identify anomalous regions in an image, but limited work has sought to develop a holistic approach that can accurately estimate perception model confidence across various sources of uncertainty. We develop a probabilistic and reconstruction-based competency estimation (PaRCE) method and compare it to existing approaches for uncertainty quantification and OOD detection. We find that our method can best distinguish between correctly classified, misclassified, and OOD samples with anomalous regions, as well as between samples with visual image modifications resulting in high, medium, and low prediction accuracy. We describe how to extend our approach for anomaly localization tasks and demonstrate the ability of our approach to distinguish between regions in an image that are familiar to the perception model from those that are unfamiliar. We find that our method generates interpretable scores that most reliably capture a holistic notion of perception model confidence.

Related papers

• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [53.79160907725975]
  We find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance.
  arXiv  Detail & Related papers  (2024-03-05T11:44:14Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• MDN-VO: Estimating Visual Odometry with Confidence [34.8860186009308]
  Visual Odometry (VO) is used in many applications including robotics and autonomous systems. We propose a deep learning-based VO model to estimate 6-DoF poses, as well as a confidence model for these estimates. Our experiments show that the proposed model exceeds state-of-the-art performance in addition to detecting failure cases.
  arXiv  Detail & Related papers  (2021-12-23T19:26:04Z)
• Deblurring via Stochastic Refinement [85.42730934561101]
  We present an alternative framework for blind deblurring based on conditional diffusion models. Our method is competitive in terms of distortion metrics such as PSNR.
  arXiv  Detail & Related papers  (2021-12-05T04:36:09Z)
• PDC-Net+: Enhanced Probabilistic Dense Correspondence Network [161.76275845530964]
  Enhanced Probabilistic Dense Correspondence Network, PDC-Net+, capable of estimating accurate dense correspondences. We develop an architecture and an enhanced training strategy tailored for robust and generalizable uncertainty prediction. Our approach obtains state-of-the-art results on multiple challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-09-28T17:56:41Z)
• Uncertainty-Aware Reliable Text Classification [21.517852608625127]
  Deep neural networks have significantly contributed to the success in predictive accuracy for classification tasks. They tend to make over-confident predictions in real-world settings, where domain shifting and out-of-distribution examples exist. We propose an inexpensive framework that adopts both auxiliary outliers and pseudo off-manifold samples to train the model with prior knowledge of a certain class.
  arXiv  Detail & Related papers  (2021-07-15T04:39:55Z)
• Uncertainty-aware Generalized Adaptive CycleGAN [44.34422859532988]
  Unpaired image-to-image translation refers to learning inter-image-domain mapping in an unsupervised manner. Existing methods often learn deterministic mappings without explicitly modelling the robustness to outliers or predictive uncertainty. We propose a novel probabilistic method called Uncertainty-aware Generalized Adaptive Cycle Consistency (UGAC)
  arXiv  Detail & Related papers  (2021-02-23T15:22:35Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• Learning Accurate Dense Correspondences and When to Trust Them [161.76275845530964]
  We aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. Our approach obtains state-of-the-art results on challenging geometric matching and optical flow datasets.
  arXiv  Detail & Related papers  (2021-01-05T18:54:11Z)
• Ramifications of Approximate Posterior Inference for Bayesian Deep Learning in Adversarial and Out-of-Distribution Settings [7.476901945542385]
  We show that Bayesian deep learning models on certain occasions marginally outperform conventional neural networks. Preliminary investigations indicate the potential inherent role of bias due to choices of initialisation, architecture or activation functions.
  arXiv  Detail & Related papers  (2020-09-03T16:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.