Recalibrating binary probabilistic classifiers

• URL: http://arxiv.org/abs/2505.19068v2
• Date: Fri, 18 Jul 2025 12:30:13 GMT
• Title: Recalibrating binary probabilistic classifiers
• Authors: Dirk Tasche,
• Abstract summary: Recalibration of binary probabilistic classifiers to a target prior probability is an important task in areas like credit risk management.<n>We analyse methods for recalibration from a distribution shift perspective. Distribution shift assumptions linked to the area under the curve are found to be useful for the design of meaningful recalibration methods.
• Score: 1.3053649021965603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recalibration of binary probabilistic classifiers to a target prior probability is an important task in areas like credit risk management. We analyse methods for recalibration from a distribution shift perspective. Distribution shift assumptions linked to the area under the curve (AUC) of a probabilistic classifier are found to be useful for the design of meaningful recalibration methods. Two new methods called parametric covariate shift with posterior drift (CSPD) and ROC-based quasi moment matching (QMM) are proposed and tested together with some other methods in an example setting. The outcomes of the test suggest that the QMM methods discussed in the paper can provide appropriately conservative results in evaluations with concave functionals like for instance risk weights functions for credit risk.

Related papers

• Doubly-Robust Estimation of Counterfactual Policy Mean Embeddings [24.07815507403025]
  Estimating the distribution of outcomes under counterfactual policies is critical for decision-making in domains such as recommendation, advertising, and healthcare.<n>We analyze a novel framework-Counterfactual Policy Mean Embedding (CPME)-that represents the entire counterfactual outcome distribution in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2025-06-03T12:16:46Z)
• Likelihood-Ratio Regularized Quantile Regression: Adapting Conformal Prediction to High-Dimensional Covariate Shifts [35.16750653336608]
  We introduce the likelihood ratio regularized quantile regression algorithm, which combines the pinball loss with a novel choice of regularization.<n>We show that the LR-QR method has coverage at the desired level in the target domain, up to a small error term.<n>Our experiments demonstrate that the LR-QR algorithm outperforms existing methods on high-dimensional prediction tasks.
  arXiv  Detail & Related papers  (2025-02-18T16:46:44Z)
• Joint empirical risk minimization for instance-dependent positive-unlabeled data [4.112909937203119]
  Learning from positive and unlabeled data (PU learning) is actively researched machine learning task. The goal is to train a binary classification model based on a dataset containing part on positives which are labeled, and unlabeled instances. Unlabeled set includes remaining part positives and all negative observations.
  arXiv  Detail & Related papers  (2023-12-27T12:45:12Z)
• Class Probability Matching Using Kernel Methods for Label Shift Adaptation [10.926835355554553]
  We propose a new framework called textitclass probability matching (textitCPM) for label shift adaptation. By incorporating the kernel logistic regression into the CPM framework to estimate the conditional probability, we propose an algorithm called textitCPMKM for label shift adaptation.
  arXiv  Detail & Related papers  (2023-12-12T13:59:37Z)
• Conformalized Unconditional Quantile Regression [27.528258690139793]
  We develop a predictive inference procedure that combines conformal prediction with unconditional quantile regression. We show that our procedure is adaptive to heteroscedasticity, provides transparent coverage guarantees that are relevant to the test instance at hand, and performs competitively with existing methods in terms of efficiency.
  arXiv  Detail & Related papers  (2023-04-04T00:20:26Z)
• Probabilistic learning constrained by realizations using a weak formulation of Fourier transform of probability measures [0.0]
  This paper deals with the taking into account a given set of realizations as constraints in the Kullback-Leibler minimum principle. A functional approach is developed on the basis of a weak formulation of the Fourier transform of probability measures. The presented application in high dimension demonstrates the efficiency and the robustness of the proposed algorithm.
  arXiv  Detail & Related papers  (2022-05-06T08:54:57Z)
• Risk Consistent Multi-Class Learning from Label Proportions [64.0125322353281]
  This study addresses a multiclass learning from label proportions (MCLLP) setting in which training instances are provided in bags. Most existing MCLLP methods impose bag-wise constraints on the prediction of instances or assign them pseudo-labels. A risk-consistent method is proposed for instance classification using the empirical risk minimization framework.
  arXiv  Detail & Related papers  (2022-03-24T03:49:04Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• Optimal variance-reduced stochastic approximation in Banach spaces [114.8734960258221]
  We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. We establish non-asymptotic bounds for both the operator defect and the estimation error.
  arXiv  Detail & Related papers  (2022-01-21T02:46:57Z)
• CC-Cert: A Probabilistic Approach to Certify General Robustness of Neural Networks [58.29502185344086]
  In safety-critical machine learning applications, it is crucial to defend models against adversarial attacks. It is important to provide provable guarantees for deep learning models against semantically meaningful input transformations. We propose a new universal probabilistic certification approach based on Chernoff-Cramer bounds.
  arXiv  Detail & Related papers  (2021-09-22T12:46:04Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.