Targeted tuning of random forests for quantile estimation and prediction intervals

• URL: http://arxiv.org/abs/2507.01430v1
• Date: Wed, 02 Jul 2025 07:32:59 GMT
• Title: Targeted tuning of random forests for quantile estimation and prediction intervals
• Authors: Matthew Berkowitz, Rachel MacKay Altman, Thomas M. Loughin,
• Abstract summary: We present a novel tuning procedure for random forests (RFs) that improves the accuracy of estimated quantiles.<n>We show that QCL tuning results in quantile estimates with more accurate coverage probabilities than those achieved using default parameter values.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a novel tuning procedure for random forests (RFs) that improves the accuracy of estimated quantiles and produces valid, relatively narrow prediction intervals. While RFs are typically used to estimate mean responses (conditional on covariates), they can also be used to estimate quantiles by estimating the full distribution of the response. However, standard approaches for building RFs often result in excessively biased quantile estimates. To reduce this bias, our proposed tuning procedure minimizes "quantile coverage loss" (QCL), which we define as the estimated bias of the marginal quantile coverage probability estimate based on the out-of-bag sample. We adapt QCL tuning to handle censored data and demonstrate its use with random survival forests. We show that QCL tuning results in quantile estimates with more accurate coverage probabilities than those achieved using default parameter values or traditional tuning (using MSPE for uncensored data and C-index for censored data), while also reducing the estimated MSE of these coverage probabilities. We discuss how the superior performance of QCL tuning is linked to its alignment with the estimation goal. Finally, we explore the validity and width of prediction intervals created using this method.

Related papers

• Multi-level Monte Carlo Dropout for Efficient Uncertainty Quantification [0.0]
  We develop multilevel Monte Carlo (MLMC) framework for uncertainty quantification.<n>Treat dropout masks as a source of epistemic randomness, we define a fidelity hierarchy by the number of forward passes used to estimate predictive moments.<n>We derive explicit bias, variance and effective cost expressions, together with sample-allocation rules across levels.
  arXiv  Detail & Related papers  (2026-01-19T18:17:25Z)
• Calibration Prediction Interval for Non-parametric Regression and Neural Networks [0.0]
  We develop a so-called calibration PI (cPI) which leverages estimations by Deep Neural Networks (DNN) or kernel methods.<n>We demonstrate that cPI based on the kernel method ensures a coverage rate with a high probability when the sample size is large.<n>A comprehensive simulation study supports the usefulness of cPI, and the convincing performance of cPI with a short sample is confirmed.
  arXiv  Detail & Related papers  (2025-09-02T18:30:39Z)
• Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees [5.09580026885155]
  We propose a frequency-based uncertainty quantification method under black-box settings.<n>Our approach involves multiple independent samplings of the model's output distribution for each input.<n>We show that frequency-based PE outperforms logit-based PE in distinguishing between correct and incorrect predictions.
  arXiv  Detail & Related papers  (2025-08-07T16:22:49Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Semiparametric conformal prediction [79.6147286161434]
  We construct a conformal prediction set accounting for the joint correlation structure of the vector-valued non-conformity scores.<n>We flexibly estimate the joint cumulative distribution function (CDF) of the scores.<n>Our method yields desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
  Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs.<n>We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint.<n>We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
  arXiv  Detail & Related papers  (2024-06-05T13:36:38Z)
• Overlapping Batch Confidence Intervals on Statistical Functionals Constructed from Time Series: Application to Quantiles, Optimization, and Estimation [5.068678962285631]
  We propose a confidence interval procedure for statistical functionals constructed using data from a stationary time series. The OBx limits, certain functionals of the Wiener process parameterized by the size of the batches and the extent of their overlap, form the essential machinery for characterizing dependence.
  arXiv  Detail & Related papers  (2023-07-17T16:21:48Z)
• Will My Robot Achieve My Goals? Predicting the Probability that an MDP Policy Reaches a User-Specified Behavior Target [56.99669411766284]
  As an autonomous system performs a task, it should maintain a calibrated estimate of the probability that it will achieve the user's goal. This paper considers settings where the user's goal is specified as a target interval for a real-valued performance summary. We compute the probability estimates by inverting conformal prediction.
  arXiv  Detail & Related papers  (2022-11-29T18:41:20Z)
• Estimation and Applications of Quantiles in Deep Binary Classification [0.0]
  Quantile regression, based on check loss, is a widely used inferential paradigm in Statistics. We consider the analogue of check loss in the binary classification setting. We develop individualized confidence scores that can be used to decide whether a prediction is reliable.
  arXiv  Detail & Related papers  (2021-02-09T07:07:42Z)
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models [80.22609163316459]
  We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost.
  arXiv  Detail & Related papers  (2020-04-01T11:49:30Z)
• Censored Quantile Regression Forest [81.9098291337097]
  We develop a new estimating equation that adapts to censoring and leads to quantile score whenever the data do not exhibit censoring. The proposed procedure named it censored quantile regression forest, allows us to estimate quantiles of time-to-event without any parametric modeling assumption.
  arXiv  Detail & Related papers  (2020-01-08T23:20:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.