U-Calibration: Forecasting for an Unknown Agent
- URL: http://arxiv.org/abs/2307.00168v1
- Date: Fri, 30 Jun 2023 23:05:26 GMT
- Title: U-Calibration: Forecasting for an Unknown Agent
- Authors: Robert Kleinberg, Renato Paes Leme, Jon Schneider, Yifeng Teng
- Abstract summary: We show that optimizing forecasts for a single scoring rule cannot guarantee low regret for all possible agents.
We present a new metric for evaluating forecasts that we call U-calibration, equal to the maximal regret of the sequence of forecasts.
- Score: 29.3181385170725
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We consider the problem of evaluating forecasts of binary events whose
predictions are consumed by rational agents who take an action in response to a
prediction, but whose utility is unknown to the forecaster. We show that
optimizing forecasts for a single scoring rule (e.g., the Brier score) cannot
guarantee low regret for all possible agents. In contrast, forecasts that are
well-calibrated guarantee that all agents incur sublinear regret. However,
calibration is not a necessary criterion here (it is possible for miscalibrated
forecasts to provide good regret guarantees for all possible agents), and
calibrated forecasting procedures have provably worse convergence rates than
forecasting procedures targeting a single scoring rule.
Motivated by this, we present a new metric for evaluating forecasts that we
call U-calibration, equal to the maximal regret of the sequence of forecasts
when evaluated under any bounded scoring rule. We show that sublinear
U-calibration error is a necessary and sufficient condition for all agents to
achieve sublinear regret guarantees. We additionally demonstrate how to compute
the U-calibration error efficiently and provide an online algorithm that
achieves $O(\sqrt{T})$ U-calibration error (on par with optimal rates for
optimizing for a single scoring rule, and bypassing lower bounds for the
traditionally calibrated learning procedures). Finally, we discuss
generalizations to the multiclass prediction setting.
Related papers
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.
Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.
We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
arXiv Detail & Related papers (2024-10-13T15:37:11Z) - Calibrated Probabilistic Forecasts for Arbitrary Sequences [58.54729945445505]
Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors.
We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves.
arXiv Detail & Related papers (2024-09-27T21:46:42Z) - Doubly Calibrated Estimator for Recommendation on Data Missing Not At
Random [20.889464448762176]
We argue that existing estimators rely on miscalibrated imputed errors and propensity scores.
We propose a Doubly Calibrated Estimator that involves the calibration of both the imputation and propensity models.
arXiv Detail & Related papers (2024-02-26T05:08:52Z) - Forecasting for Swap Regret for All Downstream Agents [5.068386855717017]
We study the problem of making predictions so that downstream agents who best respond to them will be guaranteed diminishing swap regret.
We show that by making predictions that are not calibrated, but are unbiased subject to a carefully selected collection of events, we can guarantee arbitrary downstream agents diminishing swap regret.
arXiv Detail & Related papers (2024-02-13T19:39:11Z) - Calibration by Distribution Matching: Trainable Kernel Calibration
Metrics [56.629245030893685]
We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression.
These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization.
We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
arXiv Detail & Related papers (2023-10-31T06:19:40Z) - Uncertainty Calibration for Counterfactual Propensity Estimation in Recommendation [22.67361489565711]
inverse propensity score (IPS) is employed to weight the prediction error of each observed instance.
IPS-based recommendations are hampered by miscalibration in propensity estimation.
We introduce a model-agnostic calibration framework for propensity-based debiasing of CVR predictions.
arXiv Detail & Related papers (2023-03-23T00:42:48Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Selective Regression Under Fairness Criteria [30.672082160544996]
In some cases, the performance of minority group can decrease while we reduce the coverage.
We show that such an unwanted behavior can be avoided if we can construct features satisfying the sufficiency criterion.
arXiv Detail & Related papers (2021-10-28T19:05:12Z) - Distribution-free uncertainty quantification for classification under
label shift [105.27463615756733]
We focus on uncertainty quantification (UQ) for classification problems via two avenues.
We first argue that label shift hurts UQ, by showing degradation in coverage and calibration.
We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
arXiv Detail & Related papers (2021-03-04T20:51:03Z) - Individual Calibration with Randomized Forecasting [116.2086707626651]
We show that calibration for individual samples is possible in the regression setup if the predictions are randomized.
We design a training objective to enforce individual calibration and use it to train randomized regression functions.
arXiv Detail & Related papers (2020-06-18T05:53:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.