Learning Prediction Intervals for Regression: Generalization and Calibration

• URL: http://arxiv.org/abs/2102.13625v1
• Date: Fri, 26 Feb 2021 17:55:30 GMT
• Title: Learning Prediction Intervals for Regression: Generalization and Calibration
• Authors: Haoxian Chen, Ziyi Huang, Henry Lam, Huajie Qian, Haofeng Zhang
• Abstract summary: We study the generation of prediction intervals in regression for uncertainty quantification. We use a general learning theory to characterize the optimality-feasibility tradeoff that encompasses Lipschitz continuity and VC-subgraph classes. We empirically demonstrate the strengths of our interval generation and calibration algorithms in terms of testing performances compared to existing benchmarks.
• Score: 12.576284277353606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the generation of prediction intervals in regression for uncertainty quantification. This task can be formalized as an empirical constrained optimization problem that minimizes the average interval width while maintaining the coverage accuracy across data. We strengthen the existing literature by studying two aspects of this empirical optimization. First is a general learning theory to characterize the optimality-feasibility tradeoff that encompasses Lipschitz continuity and VC-subgraph classes, which are exemplified in regression trees and neural networks. Second is a calibration machinery and the corresponding statistical theory to optimally select the regularization parameter that manages this tradeoff, which bypasses the overfitting issues in previous approaches in coverage attainment. We empirically demonstrate the strengths of our interval generation and calibration algorithms in terms of testing performances compared to existing benchmarks.

Related papers

• h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective [12.903217487071172]
  Deep neural networks have demonstrated remarkable performance across numerous learning tasks but often suffer from miscalibration.<n>This has inspired many recent works on mitigating miscalibration, particularly through post-hoc recalibration methods.<n>We propose a probabilistic learning framework for calibration called h-calibration, which theoretically constructs an equivalent learning formulation for canonical calibration with boundedness.<n>Our method not only overcomes the ten identified limitations but also achieves markedly better performance than traditional methods.
  arXiv  Detail & Related papers  (2025-06-22T09:56:44Z)
• Uniform convergence of the smooth calibration error and its relationship with functional gradient [10.906645958268939]
  This work focuses on the smooth calibration error (CE) and provides a uniform convergence bound.<n>We analyze three representative algorithms: gradient boosting trees, kernel boosting, and two-layer neural networks.<n>Our results offer new theoretical insights and practical guidance for designing reliable probabilistic models.
  arXiv  Detail & Related papers  (2025-05-26T01:23:56Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Semiparametric Counterfactual Regression [2.356908851188234]
  We propose a doubly robust-style estimator for counterfactual regression within a generalizable framework. Our approach uses incremental interventions to enhance adaptability while maintaining with standard methods. Our analysis shows that the proposed estimators can achieve $sqrn$-consistency and normality for a broad class of problems.
  arXiv  Detail & Related papers  (2025-04-03T15:32:26Z)
• Calibration Strategies for Robust Causal Estimation: Theoretical and Empirical Insights on Propensity Score Based Estimators [0.6562256987706128]
  partitioning of data for estimation and calibration critically impacts the performance of propensity score based estimators. We extend recent advances in calibration techniques for propensity score estimation, improving the robustness of propensity scores in challenging settings.
  arXiv  Detail & Related papers  (2025-03-21T16:41:10Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We show that calibration error and refinement error are not minimized simultaneously during training. We introduce a new metric for early stopping and hyper parameter tuning that makes it possible to minimize refinement error during training. Our method integrates seamlessly with any architecture and consistently improves performance across diverse classification tasks.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Calibrated Probabilistic Forecasts for Arbitrary Sequences [58.54729945445505]
  Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors. We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves.
  arXiv  Detail & Related papers  (2024-09-27T21:46:42Z)
• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• PAC-Bayes Analysis for Recalibration in Classification [4.005483185111992]
  We conduct a generalization analysis of the calibration error under the probably approximately correct (PAC) Bayes framework. We then propose a generalization-aware recalibration algorithm based on our generalization theory.
  arXiv  Detail & Related papers  (2024-06-10T12:53:13Z)
• Conformalized Survival Distributions: A Generic Post-Process to Increase Calibration [6.868842871753991]
  Discrimination and calibration represent two important properties of survival analysis. With their distinct nature, it is hard for survival models to simultaneously optimize both of them. This paper introduces a novel approach utilizing conformal regression that can improve a model's calibration without degrading discrimination.
  arXiv  Detail & Related papers  (2024-05-12T20:27:34Z)
• Multiclass Alignment of Confidence and Certainty for Network Calibration [10.15706847741555]
  Recent studies reveal that deep neural networks (DNNs) are prone to making overconfident predictions. We propose a new train-time calibration method, which features a simple, plug-and-play auxiliary loss known as multi-class alignment of predictive mean confidence and predictive certainty (MACC) Our method achieves state-of-the-art calibration performance for both in-domain and out-domain predictions.
  arXiv  Detail & Related papers  (2023-09-06T00:56:24Z)
• A Large-Scale Study of Probabilistic Calibration in Neural Network Regression [3.13468877208035]
  We conduct the largest empirical study to date to assess the probabilistic calibration of neural networks. We introduce novel differentiable recalibration and regularization methods, uncovering new insights into their effectiveness.
  arXiv  Detail & Related papers  (2023-06-05T09:33:39Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Adaptive Dimension Reduction and Variational Inference for Transductive Few-Shot Classification [2.922007656878633]
  We propose a new clustering method based on Variational Bayesian inference, further improved by Adaptive Dimension Reduction. Our proposed method significantly improves accuracy in the realistic unbalanced transductive setting on various Few-Shot benchmarks.
  arXiv  Detail & Related papers  (2022-09-18T10:29:02Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Predicting Deep Neural Network Generalization with Perturbation Response Curves [58.8755389068888]
  We propose a new framework for evaluating the generalization capabilities of trained networks. Specifically, we introduce two new measures for accurately predicting generalization gaps. We attain better predictive scores than the current state-of-the-art measures on a majority of tasks in the Predicting Generalization in Deep Learning (PGDL) NeurIPS 2020 competition.
  arXiv  Detail & Related papers  (2021-06-09T01:37:36Z)
• Privacy Preserving Recalibration under Domain Shift [119.21243107946555]
  We introduce a framework that abstracts out the properties of recalibration problems under differential privacy constraints. We also design a novel recalibration algorithm, accuracy temperature scaling, that outperforms prior work on private datasets.
  arXiv  Detail & Related papers  (2020-08-21T18:43:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.