Confidence Estimation via Auxiliary Models

• URL: http://arxiv.org/abs/2012.06508v1
• Date: Fri, 11 Dec 2020 17:21:12 GMT
• Title: Confidence Estimation via Auxiliary Models
• Authors: Charles Corbi\`ere, Nicolas Thome, Antoine Saporta, Tuan-Hung Vu, Matthieu Cord, Patrick P\'erez
• Abstract summary: We introduce a novel target criterion for model confidence, namely the true class probability ( TCP) We show that TCP offers better properties for confidence estimation than standard maximum class probability (MCP)
• Score: 47.08749569008467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reliably quantifying the confidence of deep neural classifiers is a challenging yet fundamental requirement for deploying such models in safety-critical applications. In this paper, we introduce a novel target criterion for model confidence, namely the true class probability (TCP). We show that TCP offers better properties for confidence estimation than standard maximum class probability (MCP). Since the true class is by essence unknown at test time, we propose to learn TCP criterion from data with an auxiliary model, introducing a specific learning scheme adapted to this context. We evaluate our approach on the task of failure prediction and of self-training with pseudo-labels for domain adaptation, which both necessitate effective confidence estimates. Extensive experiments are conducted for validating the relevance of the proposed approach in each task. We study various network architectures and experiment with small and large datasets for image classification and semantic segmentation. In every tested benchmark, our approach outperforms strong baselines.

Related papers

• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
  We introduce the first formal probabilistic evaluation framework in Large Language Models (LLMs) We derive novel metrics with high-probability guarantees concerning the output distribution of a model. Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
  arXiv  Detail & Related papers  (2024-10-04T15:44:23Z)
• Robust Deep Learning for Autonomous Driving [0.0]
  We introduce a new criterion to reliably estimate model confidence: the true class probability ( TCP) Since the true class is by essence unknown at test time, we propose to learn TCP criterion from data with an auxiliary model, introducing a specific learning scheme adapted to this context. We tackle the challenge of jointly detecting misclassification and out-of-distributions samples by introducing a new uncertainty measure based on evidential models and defined on the simplex.
  arXiv  Detail & Related papers  (2022-11-14T22:07:11Z)
• Using Sum-Product Networks to Assess Uncertainty in Deep Active Learning [3.7507283158673212]
  This paper proposes a new and very simple approach to computing uncertainty in deep active learning with a Convolutional Neural Network (CNN) The main idea is to use the feature representation extracted by the CNN as data for training a Sum-Product Network (SPN)
  arXiv  Detail & Related papers  (2022-06-20T14:28:19Z)
• Learning Optimal Conformal Classifiers [32.68483191509137]
  Conformal prediction (CP) is used to predict confidence sets containing the true class with a user-specified probability. This paper explores strategies to differentiate through CP during training with the goal of training model with the conformal wrapper end-to-end. We show that conformal training (ConfTr) outperforms state-of-the-art CP methods for classification by reducing the average confidence set size.
  arXiv  Detail & Related papers  (2021-10-18T11:25:33Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Failure Prediction by Confidence Estimation of Uncertainty-Aware Dirichlet Networks [6.700873164609009]
  It is shown that uncertainty-aware deep Dirichlet neural networks provide an improved separation between the confidence of correct and incorrect predictions in the true class probability metric. A new criterion is proposed for learning the true class probability by matching prediction confidence scores while taking imbalance and TCP constraints into account.
  arXiv  Detail & Related papers  (2020-10-19T21:06:45Z)
• A general framework for defining and optimizing robustness [74.67016173858497]
  We propose a rigorous and flexible framework for defining different types of robustness properties for classifiers. Our concept is based on postulates that robustness of a classifier should be considered as a property that is independent of accuracy. We develop a very general robustness framework that is applicable to any type of classification model.
  arXiv  Detail & Related papers  (2020-06-19T13:24:20Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.