Verdict Instability of OOD Scores under Reference Resampling

• URL: http://arxiv.org/abs/2609.00691v1
• Date: Tue, 01 Sep 2026 04:12:15 GMT
• Title: Verdict Instability of OOD Scores under Reference Resampling
• Abstract summary: Post-hoc out-of-distribution detectors are fitted on a finite reference set, so every score they produce is an estimate.<n>We measure verdict instability by resampling the reference set and recording the bootstrap standard deviation of the score.<n>We give a rule that predicts this sign for any score from a single label-free correlation, and abstention driven by a wrong-signed score turns out worse than abstention at random on every dataset we test.
• Score: 4.260015792110957
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Post-hoc out-of-distribution detectors are fitted on a finite reference set, so every score they produce is an estimate. If we had chosen a different set, some verdicts would have moved. We measure that movement by resampling the reference set and recording the bootstrap standard deviation of the score, which we call verdict instability. It admits a closed form with no fitted parameters. The instability of a verdict is the within-class dispersion of the assigned class along the query's direction, divided by the square root of that class's reference count. That count is what separates verdict instability from the geometry of the score distribution, and it is identifiable only under class imbalance. Instability grows with the local dispersion. Far-OOD queries lie along the low-variance directions of an anisotropic embedding, so every distance-based score we test assigns its highest values to the verdicts that are most reproducible. Only estimators of local dispersion carry the sign a practitioner expects. We give a rule that predicts this sign for any score from a single label-free correlation, and abstention driven by a wrong-signed score turns out worse than abstention at random on every dataset we test.

Related papers

• Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores [2.2400646874013037]
  Under label shift, standard conformal procedures no longer maintain their intended coverage behavior.<n>We propose the emphLabel-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive identity.<n>LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores.
  arXiv  Detail & Related papers  (2026-09-11T03:06:31Z)
• A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model [0.0]
  An external reference set is used to decompose an LLM-judge panel's error into a quality signal and a shared common-mode error.<n>We study when that assumption can be dropped and replaced by an estimate.<n>A clean-anchor estimator reports a contaminated companion anchor as fully clean once its trusted anchor is itself contaminated.
  arXiv  Detail & Related papers  (2026-09-08T14:49:13Z)
• Bias-Corrected Ceilings of Emotion Predictability from Human Label Variation Based on Instance-Level Fano Bounds [0.0]
  We propose a framework that estimates a bias-corrected ceiling, separates irreducible from reducible error, and disciplines the resulting claims.<n>Methodologically, unconstrained point estimates place reachability anywhere from 0.38 to 1.03, so saturation cannot be decided by any single estimator.
  arXiv  Detail & Related papers  (2026-08-16T08:34:46Z)
• Denoised Conformal Alignment for Reliable Selection of Conditional Average Treatment Effect Predictions [10.945117945979566]
  We study reliable selection for black-box CATE predictors.<n>We construct doubly robust proxy errors from pseudo-outcomes.<n>We propose Denoised Conformal Alignment.
  arXiv  Detail & Related papers  (2026-07-03T10:00:28Z)
• Null-Calibrated Conformal Selection via Target-Membership Scores [0.6599344783327052]
  Conformal selection aims to identify test candidates whose unknown responses fall in a target region while controlling the false discovery rate.<n>We argue that the natural score for selection is instead the target-membership probability.<n>We study membership-score-based conformal selection and isolate one conformal calibration route, Null-Calibrated Conformal Selection.
  arXiv  Detail & Related papers  (2026-06-21T05:03:04Z)
• Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection [56.362776482614976]
  Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection.<n>We show that this protocol can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space.<n>We introduce a simple training-free diagnostic, neighborhood class leakage, and show that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette.
  arXiv  Detail & Related papers  (2026-05-23T06:27:42Z)
• Empirical Bayes Conformal Prediction for Vision and Language Models [20.40115871314124]
  Conformal prediction (CP) gives distribution-free coverage for modern vision and language models.<n>Standard CP uses one realization, while average-then-calibrate variants smooth multiple realizations into a point estimate.<n>We describe an empirical Bayes conformal prediction framework that uses $r$-values to convert score variability into an uncertainty informed nonconformity score.
  arXiv  Detail & Related papers  (2026-05-22T03:17:14Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• Judging with Confidence: Calibrating Autoraters to Preference Distributions [56.17041629492863]
  We argue that a reliable autorater must learn to model the full distribution of preferences defined by a target population.<n>We present two learning methods tailored to different data conditions.<n>Our results show that finetuning autoraters with a distribution-matching objective leads to verbalized probability predictions that are better aligned with the target preference distribution.
  arXiv  Detail & Related papers  (2025-09-30T20:36:41Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• A Connection Between Learning to Reject and Bhattacharyya Divergences [57.942664964198286]
  We consider learning a joint ideal distribution over both inputs and labels.<n>We develop a link between rejection and thresholding different statistical divergences.<n>In general, we find that rejecting via a Bhattacharyya divergence is less aggressive than Chow's Rule.
  arXiv  Detail & Related papers  (2025-05-08T14:18:42Z)
• Reducing Biases in Record Matching Through Scores Calibration [1.5530839016602822]
  We propose a threshold-independent framework for measuring and reducing score bias.<n>We show that several state-of-the-art matching methods exhibit substantial score bias, even when appearing fair under standard threshold-based metrics.<n>We introduce two post-processing score calibration algorithms. The first, calib, aligns group-wise score distributions using the Wasserstein barycenter, targeting demographic parity.<n>The second, ccalib, conditions on predicted labels to further reduce label-dependent biases, such as equal opportunity.
  arXiv  Detail & Related papers  (2024-11-03T21:01:40Z)
• Correcting Underrepresentation and Intersectional Bias for Classification [49.1574468325115]
  We consider the problem of learning from data corrupted by underrepresentation bias. We show that with a small amount of unbiased data, we can efficiently estimate the group-wise drop-out rates. We show that our algorithm permits efficient learning for model classes of finite VC dimension.
  arXiv  Detail & Related papers  (2023-06-19T18:25:44Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.