When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression

• URL: http://arxiv.org/abs/2605.12780v1
• Date: Tue, 12 May 2026 21:49:11 GMT
• Title: When to Trust Confidence Thresholding: Calibration Diagnostics for Pseudo-Labelled Regression
• Abstract summary: A standard practice is to threshold the calibrated score at a confidence cutoff and treat the hard label as the truth.<n>We develop a calibration-aware diagnostic apparatus for pseudo-labelling pipelines.<n>We derive a closed-form expression for the attenuation bias that confidence thresholding induces in the downstream regression coefficient.
• Score: 1.14219428942199
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Calibrated probability outputs of trained classifiers are increasingly used as inputs to downstream regression estimands such as effects, prevalences, or disparities for a latent group observed only on a small labelled subset. A standard practice is to threshold the calibrated score at a confidence cutoff and treat the hard label as the truth. Building on a recent identification result for the underlying moment equation, we develop a calibration-aware diagnostic apparatus for pseudo-labelling pipelines. We derive a closed-form expression for the attenuation bias that confidence thresholding induces in the downstream regression coefficient, and show that the bias can be predicted, before any inference is run, from the residual score variance $V^{*}=\mathbb{E}[\operatorname{Var}(p\mid X)]$ on the unlabelled set after partialling out the downstream controls $X$. We further obtain a sharp sensitivity bound under bounded calibration drift, and identify the boundary $V^{*}=0$, which holds iff $p$ is a deterministic function of $X$; this motivates a structural separation between classifier features $W$ and downstream controls $X\subsetneq W$. Five controlled simulations and a UCI Adult illustration trace the predictions. The contribution is operational: a $(V^{*}, κ)$ decision rule that practitioners can compute from any classifier output to decide whether confidence thresholding is safe.

Related papers

• Coarsening Latent-Class Probabilities: Directional Distortion and Coverage Loss [1.14219428942199]
  We show that outcomes are increasingly regressed on a probability vector for unobserved class membership.<n>Under a constant-coefficient structural mean and conditional calibration, the observed-data problem is a partially linear regression of the outcome on the probability vector.<n>For any coarsening, the plug-in estimator converges to $mathcalA$, where the coarsening operator satisfies $mathcalA$ with $u$ the discarded signal.
  arXiv  Detail & Related papers  (2026-08-12T08:26:33Z)
• Black-Box Assisted Regression: Phase Transitions and Minimax Optimality [13.030603690976202]
  Foundation models are often used as fixed black-box predictors for downstream tasks with limited labeled data.<n>We give a finite-sample minimax characterization showing a phase transition at $_c(n) asymp n-/(2+d)$, with leading risk $min2, n-2/(2+d)$.<n>We analyze a Safe Residual Estimator: it learns a correction around $f_0$, initializes the residual head at zero so the initial predictor equals $f_0$, and uses holdout selection
  arXiv  Detail & Related papers  (2026-06-24T12:11:59Z)
• Is Spurious Correlation Removal Always Learnable? [56.28155520961125]
  Invariant learning can fail even when the structure is statistically identifiable.<n>We show a conditional computational barrier: under a black-box samplable supervised sparse recovery primitive, there exist emphamplable multi-dimensional environments.<n>Synthetic and real datasets illustrate the predicted gaps and transitions and motivate simple diversity diagnostics.
  arXiv  Detail & Related papers  (2026-06-11T05:49:43Z)
• Bandit Convex Optimization with Gradient Prediction Adaptivity [56.816177049016794]
  We investigate whether optimistic gradient predictions can improve worst-case regret guarantees in a prediction-adaptive manner.<n>The key idea is a novel variance-reduced gradient estimator whose variance scales with the prediction error rather than the gradient norm.<n>We establish an information-theoretic lower bound that scales as $(sqrtmathbbE[S_T])$, providing a fundamental characterization of the best achievable prediction-adaptive regret.
  arXiv  Detail & Related papers  (2026-05-21T08:57:38Z)
• Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements [66.94250413799232]
  We study mean estimation of a random vector $X$ in a distributed parameter-server-worker setup.<n>Key challenges are adversarial measurements and asynchrony.<n>Results provide a unified finite-time characterization of robustness, identifiability, and statistical efficiency in distributed linear estimation with adversarial workers.
  arXiv  Detail & Related papers  (2026-04-07T11:45:55Z)
• Partial Causal Structure Learning for Valid Selective Conformal Inference under Interventions [0.0]
  In genomics experiments, exchangeability often holds only within subsets of interventions that leave a target variable "unaffected"<n>Our contributions are: (i) a contamination-robust conformal coverage theorem that quantifies how misclassification of "unaffected" calibration examples degrades coverage via an explicit function $g(,n)$ of the contamination fraction and calibration set size; and (ii) a task-driven partial causal learning formulation that estimates only the binary descendant indicators $Z_a,i=mathbf1iinmathrmdesc(a)$ needed
  arXiv  Detail & Related papers  (2026-03-02T18:58:22Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Pointwise confidence estimation in the non-linear $\ell^2$-regularized least squares [12.352761060862072]
  We consider a high-probability non-asymptotic confidence estimation in the $ell2$-regularized non-linear least-squares setting with fixed design.<n>We show a pointwise confidence bound, meaning that it holds for the prediction on any given fixed test input $x$.
  arXiv  Detail & Related papers  (2025-06-08T11:23:49Z)
• Backward Conformal Prediction [46.298122008420414]
  We introduce $textitBackward Conformal Prediction$, a method that guarantees conformal coverage while providing flexible control over the size of prediction sets.<n>Our approach defines a rule that constrains how prediction set sizes behave based on the observed data, and adapts the coverage level accordingly.<n>This approach is particularly useful in applications where large prediction sets are impractical such as medical diagnosis.
  arXiv  Detail & Related papers  (2025-05-19T21:08:14Z)
• Regression-Based Estimation of Causal Effects in the Presence of Selection Bias and Confounding [52.1068936424622]
  We consider the problem of estimating the expected causal effect $E[Y|do(X)]$ for a target variable $Y$ when treatment $X$ is set by intervention.<n>In settings without selection bias or confounding, $E[Y|do(X)] = E[Y|X]$, which can be estimated using standard regression methods.<n>We propose a framework that incorporates both selection bias and confounding.
  arXiv  Detail & Related papers  (2025-03-26T13:43:37Z)
• Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration [52.70324949884702]
  We quantify the excess risk incurred using approximate posterior probabilities in batch binary decision-making.<n>We identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss.<n>On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost.
  arXiv  Detail & Related papers  (2025-03-23T10:52:36Z)
• Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making [58.06306331390586]
  We introduce the notion of a margin complement, which measures how much a prediction score $S$ changes due to a thresholding operation. We show that under suitable causal assumptions, the influences of $X$ on the prediction score $S$ are equal to the influences of $X$ on the true outcome $Y$.
  arXiv  Detail & Related papers  (2024-05-24T11:22:19Z)
• On Computationally Efficient Multi-Class Calibration [9.032290717007065]
  Project calibration gives strong guarantees for all downstream decision makers. It ensures that the probabilities predicted by summing the probabilities assigned to labels in $T$ are close to some perfectly calibrated binary predictor.
  arXiv  Detail & Related papers  (2024-02-12T17:25:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.