Decision-Making under Miscalibration

• URL: http://arxiv.org/abs/2203.09852v1
• Date: Fri, 18 Mar 2022 10:44:11 GMT
• Title: Decision-Making under Miscalibration
• Authors: Guy N. Rothblum and Gal Yona
• Abstract summary: ML-based predictions are used to inform consequential decisions about individuals. We formalize a natural (distribution-free) solution concept: given anticipated miscalibration of $alpha$, we propose using the threshold $j$ that minimizes the worst-case regret. We provide closed form expressions for $j$ when miscalibration is measured using both expected and maximum calibration error. We validate our theoretical findings on real data, demonstrating that there are natural cases in which making decisions using $j$ improves the clinical utility.
• Score: 14.762226638396209
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: ML-based predictions are used to inform consequential decisions about individuals. How should we use predictions (e.g., risk of heart attack) to inform downstream binary classification decisions (e.g., undergoing a medical procedure)? When the risk estimates are perfectly calibrated, the answer is well understood: a classification problem's cost structure induces an optimal treatment threshold $j^{\star}$. In practice, however, some amount of miscalibration is unavoidable, raising a fundamental question: how should one use potentially miscalibrated predictions to inform binary decisions? We formalize a natural (distribution-free) solution concept: given anticipated miscalibration of $\alpha$, we propose using the threshold $j$ that minimizes the worst-case regret over all $\alpha$-miscalibrated predictors, where the regret is the difference in clinical utility between using the threshold in question and using the optimal threshold in hindsight. We provide closed form expressions for $j$ when miscalibration is measured using both expected and maximum calibration error, which reveal that it indeed differs from $j^{\star}$ (the optimal threshold under perfect calibration). We validate our theoretical findings on real data, demonstrating that there are natural cases in which making decisions using $j$ improves the clinical utility.

Related papers

• Regression-Based Estimation of Causal Effects in the Presence of Selection Bias and Confounding [52.1068936424622]
  We consider the problem of estimating the expected causal effect $E[Y|do(X)]$ for a target variable $Y$ when treatment $X$ is set by intervention. In settings without selection bias or confounding, $E[Y|do(X)] = E[Y|X]$, which can be estimated using standard regression methods. We propose a framework that incorporates both selection bias and confounding.
  arXiv  Detail & Related papers  (2025-03-26T13:43:37Z)
• Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration [52.70324949884702]
  We quantify the excess risk incurred using approximate posterior probabilities in batch binary decision-making. We identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss. On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost.
  arXiv  Detail & Related papers  (2025-03-23T10:52:36Z)
• Ask for More Than Bayes Optimal: A Theory of Indecisions for Classification [1.8434042562191815]
  Selective classification is a powerful tool for automated decision-making in high-risk scenarios. Our goal is to minimize the number of indecisions, which are observations that we do not automate. By using indecisions, we are able to control the misclassification rate to any user-specified level, even below the Bayes optimal error rate.
  arXiv  Detail & Related papers  (2024-12-17T11:25:51Z)
• Towards Calibrated Losses for Adversarial Robust Reject Option Classification [3.263508275825069]
  This paper aims to characterize and design surrogates calibrated in "Adrial Robust Reject Option" setting. We provide a complete characterization result for any surrogate to be $(ell_dgamma,mathcalH_textrmlin)$- calibrated.
  arXiv  Detail & Related papers  (2024-10-14T17:17:04Z)
• Calibrating Where It Matters: Constrained Temperature Scaling [0.0]
  Clinical decision makers can use calibrated classifiers to minimise expected costs given their own cost function. We demonstrate improved calibration where it matters using convnets trained to classify dermoscopy images.
  arXiv  Detail & Related papers  (2024-06-17T12:14:31Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We prove generic upper bounds on the calibration error of any causal parameter estimate $theta$ with respect to any loss $ell$. We use our bound to analyze the convergence of two sample splitting algorithms for causal calibration.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions. We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance. Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making [58.06306331390586]
  We introduce the notion of a margin complement, which measures how much a prediction score $S$ changes due to a thresholding operation. We show that under suitable causal assumptions, the influences of $X$ on the prediction score $S$ are equal to the influences of $X$ on the true outcome $Y$.
  arXiv  Detail & Related papers  (2024-05-24T11:22:19Z)
• A Consistent and Differentiable Lp Canonical Calibration Error Estimator [21.67616079217758]
  Deep neural networks are poorly calibrated and tend to output overconfident predictions. We propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates. Our method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities.
  arXiv  Detail & Related papers  (2022-10-13T15:11:11Z)
• On Calibrated Model Uncertainty in Deep Learning [0.0]
  We extend the approximate inference for the loss-calibrated Bayesian framework to dropweights based Bayesian neural networks. We show that decisions informed by loss-calibrated uncertainty can improve diagnostic performance to a greater extent than straightforward alternatives.
  arXiv  Detail & Related papers  (2022-06-15T20:16:32Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration [118.26862029820447]
  We introduce a new notion -- emphdecision calibration -- that requires the predicted distribution and true distribution to be indistinguishable'' to a set of downstream decision-makers. Decision calibration improves decision-making on skin lesions and ImageNet classification with modern neural network.
  arXiv  Detail & Related papers  (2021-07-12T20:17:28Z)
• Recalibration of Aleatoric and Epistemic Regression Uncertainty in Medical Imaging [2.126171264016785]
  Well-calibrated uncertainty in regression allows robust rejection of unreliable predictions or detection of out-of-distribution samples. $ sigma $ scaling is able to reliably recalibrate predictive uncertainty.
  arXiv  Detail & Related papers  (2021-04-26T07:18:58Z)
• Regret-Optimal Filtering [57.51328978669528]
  We consider the problem of filtering in linear state-space models through the lens of regret optimization. We formulate a novel criterion for filter design based on the concept of regret between the estimation error energy of a clairvoyant estimator. We show that the regret-optimal estimator can be easily implemented by solving three Riccati equations and a single Lyapunov equation.
  arXiv  Detail & Related papers  (2021-01-25T19:06:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.