Improving Conditional Coverage via Orthogonal Quantile Regression
- URL: http://arxiv.org/abs/2106.00394v1
- Date: Tue, 1 Jun 2021 11:02:29 GMT
- Title: Improving Conditional Coverage via Orthogonal Quantile Regression
- Authors: Shai Feldman, Stephen Bates, Yaniv Romano
- Abstract summary: We develop a method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space.
We modify the loss function to promote independence between the size of the intervals and the indicator of a miscoverage event.
We empirically show that the modified loss function leads to improved conditional coverage, as evaluated by several metrics.
- Score: 12.826754199680472
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We develop a method to generate prediction intervals that have a
user-specified coverage level across all regions of feature-space, a property
called conditional coverage. A typical approach to this task is to estimate the
conditional quantiles with quantile regression -- it is well-known that this
leads to correct coverage in the large-sample limit, although it may not be
accurate in finite samples. We find in experiments that traditional quantile
regression can have poor conditional coverage. To remedy this, we modify the
loss function to promote independence between the size of the intervals and the
indicator of a miscoverage event. For the true conditional quantiles, these two
quantities are independent (orthogonal), so the modified loss function
continues to be valid. Moreover, we empirically show that the modified loss
function leads to improved conditional coverage, as evaluated by several
metrics. We also introduce two new metrics that check conditional coverage by
looking at the strength of the dependence between the interval size and the
indicator of miscoverage.
Related papers
- Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs.
We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint.
We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
arXiv Detail & Related papers (2024-06-05T13:36:38Z) - Integrating Uncertainty Awareness into Conformalized Quantile Regression [12.875863572064986]
We propose a new variant of the Conformalized Quantile Regression (CQR) methodology to adjust quantile regressors differentially across the feature space.
Compared to CQR, our methods enjoy the same distribution-free theoretical coverage guarantees, while demonstrating stronger conditional coverage properties in simulated settings and real-world data sets alike.
arXiv Detail & Related papers (2023-06-14T18:28:53Z) - Minimax Instrumental Variable Regression and $L_2$ Convergence
Guarantees without Identification or Closedness [71.42652863687117]
We study nonparametric estimation of instrumental variable (IV) regressions.
We propose a new penalized minimax estimator that can converge to a fixed IV solution.
We derive a strong $L$ error rate for our estimator under lax conditions.
arXiv Detail & Related papers (2023-02-10T18:08:49Z) - Will My Robot Achieve My Goals? Predicting the Probability that an MDP Policy Reaches a User-Specified Behavior Target [56.99669411766284]
As an autonomous system performs a task, it should maintain a calibrated estimate of the probability that it will achieve the user's goal.
This paper considers settings where the user's goal is specified as a target interval for a real-valued performance summary.
We compute the probability estimates by inverting conformal prediction.
arXiv Detail & Related papers (2022-11-29T18:41:20Z) - Deep Quantile Regression for Uncertainty Estimation in Unsupervised and
Supervised Lesion Detection [0.0]
Uncertainty is important in critical applications such as anomaly or lesion detection and clinical diagnosis.
In this work, we focus on using quantile regression to estimate aleatoric uncertainty and use it for estimating uncertainty in both supervised and unsupervised lesion detection problems.
We show how quantile regression can be used to characterize expert disagreement in the location of lesion boundaries.
arXiv Detail & Related papers (2021-09-20T08:50:21Z) - Understanding the Under-Coverage Bias in Uncertainty Estimation [58.03725169462616]
quantile regression tends to emphunder-cover than the desired coverage level in reality.
We prove that quantile regression suffers from an inherent under-coverage bias.
Our theory reveals that this under-coverage bias stems from a certain high-dimensional parameter estimation error.
arXiv Detail & Related papers (2021-06-10T06:11:55Z) - Optimal Adaptive Strategies for Sequential Quantum Hypothesis Testing [87.17253904965372]
We consider sequential hypothesis testing between two quantum states using adaptive and non-adaptive strategies.
We show that these errors decrease exponentially with decay rates given by the measured relative entropies between the two states.
arXiv Detail & Related papers (2021-04-30T00:52:48Z) - Universal Off-Policy Evaluation [64.02853483874334]
We take the first steps towards a universal off-policy estimator (UnO)
We use UnO for estimating and simultaneously bounding the mean, variance, quantiles/median, inter-quantile range, CVaR, and the entire cumulative distribution of returns.
arXiv Detail & Related papers (2021-04-26T18:54:31Z) - Estimation and Applications of Quantiles in Deep Binary Classification [0.0]
Quantile regression, based on check loss, is a widely used inferential paradigm in Statistics.
We consider the analogue of check loss in the binary classification setting.
We develop individualized confidence scores that can be used to decide whether a prediction is reliable.
arXiv Detail & Related papers (2021-02-09T07:07:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.