Decision-Aligned Evaluation of Uncertainty Quantification

• URL: http://arxiv.org/abs/2606.26990v1
• Date: Thu, 25 Jun 2026 13:05:41 GMT
• Title: Decision-Aligned Evaluation of Uncertainty Quantification
• Abstract summary: We introduce decision-alignment, a criterion that reveals which evaluation metrics meaningfully align with downstream utilities.<n>We propose prior-weighted utility metrics, a special class of proper scoring rules that provides decision-aligned uncertainty evaluation.
• Score: 5.535961659952622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Uncertainty estimates in machine learning are typically evaluated using generic metrics such as the negative log-likelihood and expected calibration error, yet good performance on such metrics does not necessarily imply high utility in downstream decisions. We introduce decision-alignment, a criterion that reveals which evaluation metrics meaningfully align with downstream utilities. Applying this framework, we show that many widely used uncertainty metrics are either misaligned with common decision problems or encode pathological prior beliefs about the downstream task. We then propose prior-weighted utility metrics, a special class of proper scoring rules that provides decision-aligned uncertainty evaluation. Across benchmark experiments and real-world case studies, our metrics consistently align with realized decision utility, while conventional metrics do not. Our results surface flaws in the current UQ evaluation protocol and offer a principled extension of existing metrics toward decision-relevant UQ evaluation.

Related papers

• ECUAS$_n$: A family of metrics for principled evaluation of uncertainty-augmented systems [22.79149311211994]
  In high-stakes automated decision-making, access to predictive uncertainty is essential.<n>We propose a novel family of metrics, ECUAS$_n$, formulated as proper scoring rules for the task of interest.
  arXiv  Detail & Related papers  (2026-05-19T20:55:41Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection [56.302586730134806]
  We introduce Confidence-Consistency Evaluation (CCE), a novel evaluation metric.<n>CCE simultaneously measures prediction confidence and uncertainty consistency.<n>We also establish RankEval, a benchmark for comparing the ranking capabilities of various metrics.
  arXiv  Detail & Related papers  (2025-09-01T03:38:38Z)
• A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding [61.86327960322782]
  We propose a principled framework for randomized decision-making based on interval estimates of the quality of each item.<n>We introduce MERIT, an optimization-based method that maximizes the worst-case expected number of top candidates selected.<n>We prove that MERIT satisfies desirable axiomatic properties not guaranteed by existing approaches.
  arXiv  Detail & Related papers  (2025-06-23T19:59:30Z)
• UNCERTAINTY-LINE: Length-Invariant Estimation of Uncertainty for Large Language Models [51.53270695871237]
  We show that UNCERTAINTY-LINE: consistently improves over even nominally length-normalized UQ methods uncertainty estimates.<n>Our method is post-hoc, model-agnostic, and applicable to a range of UQ measures.
  arXiv  Detail & Related papers  (2025-05-25T09:30:43Z)
• SConU: Selective Conformal Uncertainty in Large Language Models [59.25881667640868]
  We propose a novel approach termed Selective Conformal Uncertainty (SConU)<n>We develop two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level.<n>Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions.
  arXiv  Detail & Related papers  (2025-04-19T03:01:45Z)
• A Consequentialist Critique of Binary Classification Evaluation Practices [4.603739046972463]
  We show a preference for top-K metrics or fixed thresholds in evaluations at major conferences like ICML, FAccT, and CHIL.<n>We use this decision-theoretic framework to map evaluation metrics to their optimal use cases, along with a Python package, briertools, to promote the broader adoption of Brier scores.
  arXiv  Detail & Related papers  (2025-04-06T15:58:01Z)
• Legitimate ground-truth-free metrics for deep uncertainty classification scoring [3.9599054392856483]
  The use of Uncertainty Quantification (UQ) methods in production remains limited.<n>This limitation is exacerbated by the challenge of validating UQ methods in absence of UQ ground truth.<n>This paper investigates such metrics and proves that they are theoretically well-behaved and actually tied to some uncertainty ground truth.
  arXiv  Detail & Related papers  (2024-10-30T14:14:32Z)
• Towards Clear Expectations for Uncertainty Estimation [64.20262246029286]
  Uncertainty Quantification (UQ) is crucial to achieve trustworthy Machine Learning (ML) Most UQ methods suffer from disparate and inconsistent evaluation protocols. This opinion paper offers a new perspective by specifying those requirements through five downstream tasks.
  arXiv  Detail & Related papers  (2022-07-27T07:50:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.