Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions

• URL: http://arxiv.org/abs/2603.22988v1
• Date: Tue, 24 Mar 2026 09:31:13 GMT
• Title: Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions
• Abstract summary: We consider two approaches for assessing the reliability of individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ)<n>We show that RQ is capable of outperforming UQ, both in a standard setting and in the presence of distribution shift.<n>We also demonstrate the complementarity of RQ and UQ by showing that a combination of both approaches can lead to even better reliability assessments.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider two approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We explain the conceptual differences between the two approaches, compare both approaches on a number of benchmark datasets and show that RQ is capable of outperforming UQ, both in a standard setting and in the presence of distribution shift. Beside showing that RQ can be competitive with UQ, we also demonstrate the complementarity of RQ and UQ by showing that a combination of both approaches can lead to even better reliability assessments.

Related papers

• Understanding Software Defect Prediction: A Large-scale Empirical Study Across Uncertainty Quantification and Performance Evaluation [12.540031438888716]
  Software defect prediction (SDP) classifiers produce probabilities used for inspection prioritization, threshold tuning, and risk communication.<n>We conducted a large-scale empirical study of probability-based uncertainty quantification (UQ) for SDP.<n>We evaluated five UQ metrics, six performance metrics, and three calibration metrics for 16 representative classifiers.
  arXiv  Detail & Related papers  (2026-07-02T08:05:27Z)
• ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking [25.561100902733326]
  We introduce ME-IQA, a test-time memory-enhanced re-ranking framework.<n>It builds a memory bank and retrieves semantically and perceptually aligned neighbors using reasoning summaries.<n>It performs gated reflection and consolidates memory to improve future decisions.
  arXiv  Detail & Related papers  (2026-03-21T12:23:08Z)
• Robustness and uncertainty: two complementary aspects of the reliability of the predictions of a classifier [0.0]
  We consider two approaches for assessing the reliability of individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ)<n>We compare both approaches on a number of benchmark datasets and show that there is no clear winner between the two, but that they are complementary and can be combined to obtain a hybrid approach that outperforms both RQ and UQ.
  arXiv  Detail & Related papers  (2025-12-17T14:40:14Z)
• Confidence-guided Refinement Reasoning for Zero-shot Question Answering [19.796455166690187]
  Confidence-guided Refinement Reasoning (C2R) is a novel training-free framework applicable to question-answering tasks across text, image, and video domains.<n>C2R strategically constructs and refines sub-questions and their answers (sub-QAs), deriving a better confidence score for the target answer.
  arXiv  Detail & Related papers  (2025-09-25T05:15:12Z)
• Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistency [66.96286531087549]
  Uncertainty quantification (UQ) methods for Large Language Models (LLMs) encompass a variety of approaches.<n>We propose a novel approach to integrating model confidence with output consistency, resulting in a family of efficient and robust UQ methods.<n>We evaluate our approach across various tasks such as question answering, abstractive summarization, and machine translation.
  arXiv  Detail & Related papers  (2025-02-07T14:30:12Z)
• Addressing Uncertainty in LLMs to Enhance Reliability in Generative AI [47.64301863399763]
  We present a dynamic semantic clustering approach inspired by the Chinese Restaurant Process. We quantify uncertainty of Large Language Models (LLMs) on a given query by calculating entropy of the generated semantic clusters. We propose leveraging the (negative) likelihood of these clusters as the (non)conformity score within Conformal Prediction framework.
  arXiv  Detail & Related papers  (2024-11-04T18:49:46Z)
• Toward Unsupervised Realistic Visual Question Answering [70.67698100148414]
  We study the problem of realistic VQA (RVQA), where a model has to reject unanswerable questions (UQs) and answer answerable ones (AQs) We first point out 2 drawbacks in current RVQA research, where (1) datasets contain too many unchallenging UQs and (2) a large number of annotated UQs are required for training. We propose a new testing dataset, RGQA, which combines AQs from an existing VQA dataset with around 29K human-annotated UQs. This combines pseudo UQs obtained by randomly pairing images and questions, with an
  arXiv  Detail & Related papers  (2023-03-09T06:58:29Z)
• Towards Clear Expectations for Uncertainty Estimation [64.20262246029286]
  Uncertainty Quantification (UQ) is crucial to achieve trustworthy Machine Learning (ML) Most UQ methods suffer from disparate and inconsistent evaluation protocols. This opinion paper offers a new perspective by specifying those requirements through five downstream tasks.
  arXiv  Detail & Related papers  (2022-07-27T07:50:57Z)
• Counterfactual Variable Control for Robust and Interpretable Question Answering [57.25261576239862]
  Deep neural network based question answering (QA) models are neither robust nor explainable in many cases. In this paper, we inspect such spurious "capability" of QA models using causal inference. We propose a novel approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlation.
  arXiv  Detail & Related papers  (2020-10-12T10:09:05Z)
• Cross Learning in Deep Q-Networks [82.20059754270302]
  We propose a novel cross Q-learning algorithm, aim at alleviating the well-known overestimation problem in value-based reinforcement learning methods. Our algorithm builds on double Q-learning, by maintaining a set of parallel models and estimate the Q-value based on a randomly selected network.
  arXiv  Detail & Related papers  (2020-09-29T04:58:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.