ECUAS$_n$: A family of metrics for principled evaluation of uncertainty-augmented systems

• URL: http://arxiv.org/abs/2605.20490v3
• Date: Fri, 22 May 2026 20:08:09 GMT
• Title: ECUAS$_n$: A family of metrics for principled evaluation of uncertainty-augmented systems
• Abstract summary: In high-stakes automated decision-making, access to predictive uncertainty is essential.<n>We propose a novel family of metrics, ECUAS$_n$, formulated as proper scoring rules for the task of interest.
• Score: 22.79149311211994
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept or reject predictions based on application-specific cost trade-offs. Such uncertainty-augmented (UA) systems -- i.e., systems that output both predictions and uncertainty scores -- are currently being assessed in the literature in a variety of ways, using separate metrics to evaluate the predictions and the uncertainty scores, setting a cost function with a fixed rejection cost or integrating over a coverage-risk curve. We argue that these evaluation approaches are inadequate for assessing overall performance of the UA system for decision making under uncertainty and propose a novel family of metrics, ECUAS$_n$, formulated as proper scoring rules for the task of interest. The parameter $n$ controls the trade-off between the cost of incorrect predictions and imperfect uncertainties depending on the needs of the use-case. We demonstrate the advantages of the ECUAS$_n$ metrics both theoretically and empirically, through experiments on diverse classification and generation datasets, including a manually annotated subset of TriviaQA.

Related papers

• Decision-Aligned Evaluation of Uncertainty Quantification [5.535961659952622]
  We introduce decision-alignment, a criterion that reveals which evaluation metrics meaningfully align with downstream utilities.<n>We propose prior-weighted utility metrics, a special class of proper scoring rules that provides decision-aligned uncertainty evaluation.
  arXiv  Detail & Related papers  (2026-06-25T13:05:41Z)
• Rethinking XAI Evaluation: A Human-Centered Audit of Shapley Benchmarks in High-Stakes Settings [8.934085986179019]
  Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape.<n>We use a unified amortized framework to isolate semantic differences between eight Shapley variants.<n>Our results reveal a fundamental misalignment: standard quantitative metrics, such as sparsity and faithfulness, are decoupled from human-perceived clarity and decision utility.
  arXiv  Detail & Related papers  (2026-04-24T15:38:44Z)
• Fairness under uncertainty in sequential decisions [14.392931077157906]
  Algorithmic approaches alone cannot resolve structural inequalities.<n>Many real machine learning applications are online and sequential, with prior decisions informing future ones.<n>This paper introduces a taxonomy of uncertainty in sequential decision-making.
  arXiv  Detail & Related papers  (2026-04-23T14:17:58Z)
• Entropy Alone is Insufficient for Safe Selective Prediction in LLMs [20.664633053172327]
  Selective prediction systems can mitigate harms resulting from language model hallucinations by abstaining from answering in high-risk cases.<n>Uncertainty quantification techniques are often employed to identify such cases, but are rarely evaluated in the context of the wider selective prediction policy.<n>We identify a model-dependent failure mode of entropy-based uncertainty methods that leads to unreliable abstention behaviour, and address it by combining entropy scores with a correctness probe signal.
  arXiv  Detail & Related papers  (2026-03-22T11:27:13Z)
• Statistical Inference for Score Decompositions [0.0]
  We introduce methods for score decompositions, which partition scoring functions for predictive assessment into three interpretable components.<n>We find that for survey inflation forecasts, discrimination abilities can differ significantly even when overall predictive ability does not.<n>By disentangling forecast accuracy from backtest performance, the method exposes critical shortcomings in current banking regulation.
  arXiv  Detail & Related papers  (2026-03-04T16:57:08Z)
• Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation [20.726685669562496]
  Hallucinations are a common issue that undermine the reliability of large language models (LLMs)<n>Recent studies have identified a subset of hallucinations, known as confabulations, which arise due to predictive uncertainty of LLMs.<n>To detect confabulations, various methods for estimating predictive uncertainty in natural language generation (NLG) have been developed.
  arXiv  Detail & Related papers  (2025-10-02T17:54:09Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• SConU: Selective Conformal Uncertainty in Large Language Models [59.25881667640868]
  We propose a novel approach termed Selective Conformal Uncertainty (SConU)<n>We develop two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level.<n>Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions.
  arXiv  Detail & Related papers  (2025-04-19T03:01:45Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Debiasing Recommendation by Learning Identifiable Latent Confounders [49.16119112336605]
  Confounding bias arises due to the presence of unmeasured variables that can affect both a user's exposure and feedback. Existing methods either (1) make untenable assumptions about these unmeasured variables or (2) directly infer latent confounders from users' exposure. We propose a novel method, i.e., identifiable deconfounder (iDCF), which leverages a set of proxy variables to resolve the aforementioned non-identification issue.
  arXiv  Detail & Related papers  (2023-02-10T05:10:26Z)
• Robust Design and Evaluation of Predictive Algorithms under Unobserved Confounding [6.925076885159747]
  We propose a framework for robust design and evaluation of predictive algorithms.<n>We show that varying confounding assumptions substantially affects credit risk predictions and fairness evaluations across income groups.
  arXiv  Detail & Related papers  (2022-12-19T20:41:44Z)
• Uncertainty-Driven Action Quality Assessment [11.958132175629368]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores.<n>We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss.<n>Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)
• An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis [68.8204255655161]
  We show that uncertainty measures based on Monte-Carlo dropout in the context of a human-in-the-loop system increase the system's transparency and performance. A simulation study demonstrates that the uncertainty-based human-in-the-loop system increases performance for different levels of human involvement.
  arXiv  Detail & Related papers  (2020-07-14T15:47:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.