Assessing model calibration with boosting trees

• URL: http://arxiv.org/abs/2606.08084v1
• Date: Sat, 06 Jun 2026 10:14:36 GMT
• Title: Assessing model calibration with boosting trees
• Abstract summary: We show that boosting trees can be used to test necessary conditions for calibration and auto-calibration.<n>The proposed tests prove to be very powerful on a large insurance dataset.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The main goal in regression modelling consists in approximating the conditional mean of a response given a set of features. A regression function is said to be calibrated if the resulting mean estimates match the true conditional means for almost every set of features. Aiming for calibration seems not achievable in practice as one typically deals with finite samples of noisy observations. A weaker notion of calibration is auto-calibration, and it means that the expectation of responses being given the same mean estimate matches this estimate. This notion is important, e.g., in insurance pricing as it ensures no cross-subsidization between different price cohorts. In this paper, we show that boosting trees can be used to test necessary conditions for calibration and auto-calibration, respectively. The practical relevance of our approach is supported by a numerical example, in which the proposed tests prove to be very powerful on a large insurance dataset.

Related papers

• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Calibration Bands for Mean Estimates within the Exponential Dispersion Family [0.0]
  Testing for auto-calibration has only been considered recently in the literature.<n>We show that the obtained calibration bands allow us to construct various tests for calibration and auto-calibration.
  arXiv  Detail & Related papers  (2025-03-24T17:09:19Z)
• Calibration through the Lens of Interpretability [3.9962751777898955]
  calibration is a frequently invoked concept when useful label probability estimates are required on top of classification accuracy.<n>In this work, we initiate an axiomatic study of the notion of calibration.<n>We catalogue desirable properties of calibrated models as well as corresponding evaluation metrics and analyze their feasibility and correspondences.
  arXiv  Detail & Related papers  (2024-12-01T19:28:16Z)
• Measuring training variability from stochastic optimization using robust nonparametric testing [5.519968037738177]
  We propose a robust hypothesis testing framework and a novel summary statistic, the $alpha$-trimming level, to measure model similarity.<n>Applying hypothesis testing directly with the $alpha$-trimming level is challenging because we cannot accurately describe the distribution under the null hypothesis.<n>We show how to use the $alpha$-trimming level to measure model variability and demonstrate experimentally that it is more expressive than performance metrics.
  arXiv  Detail & Related papers  (2024-06-12T15:08:15Z)
• Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
  Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs.<n>We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint.<n>We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
  arXiv  Detail & Related papers  (2024-06-05T13:36:38Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• Calibration tests beyond classification [30.616624345970973]
  Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets. Calibrated models guarantee that the predictions are neither over- nor under-confident.
  arXiv  Detail & Related papers  (2022-10-21T09:49:57Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Should Ensemble Members Be Calibrated? [16.331175260764]
  Modern deep neural networks are often observed to be poorly calibrated. Deep learning approaches make use of large numbers of model parameters. This paper explores the application of calibration schemes to deep ensembles.
  arXiv  Detail & Related papers  (2021-01-13T23:59:00Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.