Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters

• URL: http://arxiv.org/abs/2607.11653v1
• Date: Mon, 13 Jul 2026 15:04:24 GMT
• Title: Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters
• Abstract summary: Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs.<n>Forecasters can look to an auditor with coarse information while being miscalibrated to an auditor with richer information.<n>We develop a distribution-free and game-theoretic testing framework for continuously auditing black-box conditional quantile forecasters with non-i.i.d. losses.
• Score: 4.458834881246723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Black-box conditional quantile forecasts are widely used for sequential decisions under asymmetric costs, such as inventory planning in supply chain management. Once deployed, such forecasters must be monitored continuously as data streams drift and regimes change; this invalidates standard, fixed-horizon backtests for calibration. Further, existing backtests do not take into account that the notion of calibration is, in fact, information-dependent: forecasts can look calibrated to an auditor with coarse information while being miscalibrated to an auditor with richer information. We develop a distribution-free and game-theoretic testing framework for continuously auditing black-box conditional quantile forecasters with non-i.i.d. losses, such that the resulting evidence process is powerful against predictably chosen alternatives specified by the features available to the auditor. We first formalize notions of conditional quantile calibration when different sets of features are available to the auditor, establishing that the coarseness of the auditor's information set determines the hardness of the testing problem. We then identify the sets of alternatives for which the auditor can achieve power, and focusing on contextual bets linear in the features, we derive finite-time detection guarantees for such alternatives, all without an i.i.d. assumption. The resulting evidence processes are interpretable at the feature level, as they quantify fine-grained, "feature-aware" evidence for miscalibration. We empirically validate these methods on simulated and real data, finding that a popular time series forecaster (Chronos-2) is highly miscalibrated w.r.t. multiple relevant features.

Related papers

• Bayesian control for coding agents [63.64172141184361]
  We propose a cost-sensitive sequential hypothesis testing framework for coding agents.<n>A Bayesian controller maintains a belief over correctness and decides whether to gather more evidence, refine the candidate, verify it, or stop.<n>We show that the belief state yields an interpretable correctness score that outperforms token-probability and raw tool-success baselines for uncertainty quantification.
  arXiv  Detail & Related papers  (2026-06-23T11:41:32Z)
• Recipes for Calibration Checks in Safety-Critical Applications [0.0]
  We introduce a framework for calibration checks for safety-critical prediction systems.<n>The checks produce a single accept/reject decision for data collected from a forecaster.<n>We demonstrate the applicability of the framework on two complementary example problems, weather forecasting and robot pose estimation.
  arXiv  Detail & Related papers  (2026-04-29T09:43:55Z)
• Statistical Inference for Score Decompositions [0.0]
  We introduce methods for score decompositions, which partition scoring functions for predictive assessment into three interpretable components.<n>We find that for survey inflation forecasts, discrimination abilities can differ significantly even when overall predictive ability does not.<n>By disentangling forecast accuracy from backtest performance, the method exposes critical shortcomings in current banking regulation.
  arXiv  Detail & Related papers  (2026-03-04T16:57:08Z)
• Calibrating an Imperfect Auxiliary Predictor for Unobserved No-Purchase Choice [1.5484595752241122]
  Firms typically cannot observe key consumer actions: whether customers buy from a competitor, choose not to buy, or even fully consider the firm's offer.<n>This missing outside-option information makes market-size and preference estimation difficult even in simple multinomial logit (MNL) models.<n>We study a complementary setting in which a black-box auxiliary predictor provides outside-option probabilities, but is potentially biased or miscalibrated because it was trained in a different channel, period, or population.<n>We develop calibration methods that turn such imperfect predictions into statistically valid no-purchase estimates using purchase-only data from the focal
  arXiv  Detail & Related papers  (2026-02-12T03:00:36Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.<n>Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.<n>We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• A Review of Uncertainty Calibration in Pretrained Object Detectors [5.440028715314566]
  We investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation. We deliver novel insights into why poor detector calibration emerges.
  arXiv  Detail & Related papers  (2022-10-06T14:06:36Z)
• Distribution-Free, Risk-Controlling Prediction Sets [112.9186453405701]
  We show how to generate set-valued predictions from a black-box predictor that control the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets.
  arXiv  Detail & Related papers  (2021-01-07T18:59:33Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.