Human-Aligned Calibration for AI-Assisted Decision Making

• URL: http://arxiv.org/abs/2306.00074v4
• Date: Fri, 23 Feb 2024 13:35:04 GMT
• Title: Human-Aligned Calibration for AI-Assisted Decision Making
• Authors: Nina L. Corvelo Benz and Manuel Gomez Rodriguez
• Abstract summary: We show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy. We show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment.
• Score: 19.767213234234855
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Whenever a binary classifier is used to provide decision support, it typically provides both a label prediction and a confidence value. Then, the decision maker is supposed to use the confidence value to calibrate how much to trust the prediction. In this context, it has been often argued that the confidence value should correspond to a well calibrated estimate of the probability that the predicted label matches the ground truth label. However, multiple lines of empirical evidence suggest that decision makers have difficulties at developing a good sense on when to trust a prediction using these confidence values. In this paper, our goal is first to understand why and then investigate how to construct more useful confidence values. We first argue that, for a broad class of utility functions, there exist data distributions for which a rational decision maker is, in general, unlikely to discover the optimal decision policy using the above confidence values -- an optimal decision maker would need to sometimes place more (less) trust on predictions with lower (higher) confidence values. However, we then show that, if the confidence values satisfy a natural alignment property with respect to the decision maker's confidence on her own predictions, there always exists an optimal decision policy under which the level of trust the decision maker would need to place on predictions is monotone on the confidence values, facilitating its discoverability. Further, we show that multicalibration with respect to the decision maker's confidence on her own predictions is a sufficient condition for alignment. Experiments on four different AI-assisted decision making tasks where a classifier provides decision support to real human experts validate our theoretical results and suggest that alignment may lead to better decisions.

Related papers

• A Confidence-Variance Theory for Pseudo-Label Selection in Semi-Supervised Learning [15.149171763610662]
  This paper introduces a Confidence-Variance (CoVar) theory framework that provides a principled joint reliability criterion for pseudo-label selection.<n>We show that reliable pseudo-labels should have both high MC and low RCV, and that the influence of RCV increases as confidence grows.<n>We integrate CoVar as a plug-in module into representative semi-supervised semantic segmentation and image classification methods.
  arXiv  Detail & Related papers  (2026-01-16T02:51:59Z)
• Optimal Conformal Prediction, E-values, Fuzzy Prediction Sets and Subsequent Decisions [0.0]
  We make three contributions to conformal prediction.<n>First, we propose fuzzy conformal confidence sets that offer a degree of exclusion.<n>Second, we derive optimal conformal confidence sets.<n>Third, we generalize the inheritance of guarantees by subsequent minimax decisions from confidence sets to fuzzy confidence sets.
  arXiv  Detail & Related papers  (2025-09-16T14:46:18Z)
• Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation [63.49409574310576]
  Large language models (LLMs) exhibit overconfidence, assigning high confidence scores to incorrect predictions.<n>We introduce FineCE, a novel confidence estimation method that delivers accurate, fine-grained confidence scores during text generation.<n>Our code and all baselines used in the paper are available on GitHub.
  arXiv  Detail & Related papers  (2025-08-16T13:29:35Z)
• Truthful Elicitation of Imprecise Forecasts [11.153198087930756]
  We propose a framework for scoring imprecise forecasts -- forecasts given as a set of beliefs. We show that truthful elicitation of imprecise forecasts is achievable using proper scoring rules randomized over the aggregation procedure.
  arXiv  Detail & Related papers  (2025-03-20T17:53:35Z)
• Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents [24.938391962245877]
  We develop decision-theoretic foundations that connect uncertainty using prediction sets with risk-averse decision-making. We experimentally demonstrate the significant advantages of Risk-Averse (RAC) in applications such as medical diagnosis and recommendation systems.
  arXiv  Detail & Related papers  (2025-02-04T18:37:10Z)
• Language Models Prefer What They Know: Relative Confidence Estimation via Confidence Preferences [62.52739672949452]
  Language models (LMs) should provide reliable confidence estimates to help users detect mistakes in their outputs and defer to human experts when necessary. We propose relative confidence estimation, where we match up questions against each other and ask the model to make relative judgments of confidence. Treating each question as a "player" in a series of matchups against other questions and the model's preferences as match outcomes, we can use rank aggregation methods like Elo rating and Bradley-Terry to translate the model's confidence preferences into confidence scores.
  arXiv  Detail & Related papers  (2025-02-03T07:43:27Z)
• Beyond Uncertainty Quantification: Learning Uncertainty for Trust-Informed Neural Network Decisions - A Case Study in COVID-19 Classification [7.383605511698832]
  Reliable uncertainty quantification is critical in high-stakes applications, such as medical diagnosis.<n>Traditional uncertainty quantification methods rely on a predefined confidence threshold to classify predictions as confident or uncertain.<n>This approach assumes that predictions exceeding the threshold are trustworthy, while those below it are uncertain, without explicitly assessing the correctness of high-confidence predictions.<n>This study proposes an uncertainty-aware stacked neural network, which extends conventional uncertainty quantification by learning when predictions should be trusted.
  arXiv  Detail & Related papers  (2024-09-19T04:20:12Z)
• Automated Trustworthiness Testing for Machine Learning Classifiers [3.3423762257383207]
  This paper proposes TOWER, the first technique to automatically create trustworthiness oracles that determine whether text classifier predictions are trustworthy. Our hypothesis is that a prediction is trustworthy if the words in its explanation are semantically related to the predicted class. The results show that TOWER can detect a decrease in trustworthiness as noise increases, but is not effective when evaluated against the human-labeled dataset.
  arXiv  Detail & Related papers  (2024-06-07T20:25:05Z)
• BayesJudge: Bayesian Kernel Language Modelling with Confidence Uncertainty in Legal Judgment Prediction [14.672477787408887]
  We present a novel approach called BayesJudge that harnesses the synergy between deep learning and deep Gaussian Processes to quantify uncertainty. Our method leverages informative priors and flexible data modelling via kernels, surpassing existing methods in both predictive accuracy and confidence estimation. We also introduce an optimal solution to automate the scrutiny of unreliable predictions, resulting in a significant increase in the accuracy of the model's predictions by up to 27%.
  arXiv  Detail & Related papers  (2024-04-16T11:42:06Z)
• U-Trustworthy Models.Reliability, Competence, and Confidence in Decision-Making [0.21756081703275998]
  We present a precise mathematical definition of trustworthiness, termed $mathcalU$-trustworthiness. Within the context of $mathcalU$-trustworthiness, we prove that properly-ranked models are inherently $mathcalU$-trustworthy. We advocate for the adoption of the AUC metric as the preferred measure of trustworthiness.
  arXiv  Detail & Related papers  (2024-01-04T04:58:02Z)
• A Diachronic Perspective on User Trust in AI under Uncertainty [52.44939679369428]
  Modern NLP systems are often uncalibrated, resulting in confidently incorrect predictions that undermine user trust. We study the evolution of user trust in response to trust-eroding events using a betting game.
  arXiv  Detail & Related papers  (2023-10-20T14:41:46Z)
• Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions [80.34972679938483]
  We introduce Conformal Decision Theory, a framework for producing safe autonomous decisions despite imperfect machine learning predictions. Decisions produced by our algorithms are safe in the sense that they come with provable statistical guarantees of having low risk. Experiments demonstrate the utility of our approach in robot motion planning around humans, automated stock trading, and robot manufacturing.
  arXiv  Detail & Related papers  (2023-10-09T17:59:30Z)
• Calibrating AI Models for Wireless Communications via Conformal Prediction [55.47458839587949]
  Conformal prediction is applied for the first time to the design of AI for communication systems. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees.
  arXiv  Detail & Related papers  (2022-12-15T12:52:23Z)
• Reliability-Aware Prediction via Uncertainty Learning for Person Image Retrieval [51.83967175585896]
  UAL aims at providing reliability-aware predictions by considering data uncertainty and model uncertainty simultaneously. Data uncertainty captures the noise" inherent in the sample, while model uncertainty depicts the model's confidence in the sample's prediction.
  arXiv  Detail & Related papers  (2022-10-24T17:53:20Z)
• Stability of Weighted Majority Voting under Estimated Weights [16.804588631149393]
  A (machine learning) algorithm that computes trust is called unbiased when it does not systematically overestimate or underestimate the trustworthiness. We introduce and analyse two important properties of such unbiased trust values: stability of correctness and stability of optimality.
  arXiv  Detail & Related papers  (2022-07-13T10:55:41Z)
• What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience [63.75363908696257]
  computational reinforcement learning seeks to construct an agent's perception of the world through predictions of future sensations. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. We introduce a meta-gradient descent process by which an agent learns what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward.
  arXiv  Detail & Related papers  (2022-06-13T21:31:06Z)
• MACEst: The reliable and trustworthy Model Agnostic Confidence Estimator [0.17188280334580192]
  We argue that any confidence estimates based upon standard machine learning point prediction algorithms are fundamentally flawed. We present MACEst, a Model Agnostic Confidence Estimator, which provides reliable and trustworthy confidence estimates.
  arXiv  Detail & Related papers  (2021-09-02T14:34:06Z)
• Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making [53.62514158534574]
  We study whether features that reveal case-specific model information can calibrate trust and improve the joint performance of the human and AI. We show that confidence score can help calibrate people's trust in an AI model, but trust calibration alone is not sufficient to improve AI-assisted decision making.
  arXiv  Detail & Related papers  (2020-01-07T15:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.