A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model

• URL: http://arxiv.org/abs/2609.08826v1
• Date: Tue, 08 Sep 2026 14:49:13 GMT
• Title: A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model
• Abstract summary: An external reference set is used to decompose an LLM-judge panel's error into a quality signal and a shared common-mode error.<n>We study when that assumption can be dropped and replaced by an estimate.<n>A clean-anchor estimator reports a contaminated companion anchor as fully clean once its trusted anchor is itself contaminated.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When an external reference set (an anchor) is used to decompose an LLM-judge panel's error into a quality signal and a shared common-mode error, standard practice assumes the anchor is uncontaminated: its error uncorrelated with the judges' shared error. We study when that assumption can be dropped and replaced by an estimate. Under a single-common-factor model, >=2 judges and >=2 anchors point-identify the quality variance, the common-mode variance, and each anchor's contamination correlation rho_k in closed form, with an exact per-anchor-pair failure boundary; a designated clean-anchor estimator, by contrast, reports a contaminated companion anchor as fully clean once its trusted anchor is itself contaminated. Because the single-common-factor assumption is itself untestable, the estimator ships gated behind a calibrated diagnostic battery (judge-covariance dispersion; over-identification; a family-block test from judge metadata, with a family-blocked estimator that removes family-level shared-residual bias exactly), bootstrap confidence intervals with measured coverage, and a weak-identification screen. A proposition maps which violations bias rho_k, in which direction, and which evade detection. For ordinal scores we show an identification hierarchy: with all variables ordinal, rho_k is not identified at any number of anchors; with ordinal judges and >=3 continuous anchors it is, and we give an estimator for that case. On real data the validation is asymmetric, and we say so plainly: the diagnostics are validated in the rejecting direction (both real panels we test are correctly rejected by the model-adequacy pre-test), while the estimator is validated in simulation and stress-tested semi-synthetically under oracle calibration; no real panel has yet passed the pre-test, and the pre-test exists precisely to say so. All results replay offline from shipped, checksummed artifacts.

Related papers

• A distribution-free certification framework for trustworthy crash-severity prediction [1.089614199781423]
  We develop a certification layer that wraps any severity model unmodified.<n>On 5.2 million Texas records across seven base models spanning four decades, the layer attaches identical validity.<n>The framework is released as an open-source package with theorem-level tests.
  arXiv  Detail & Related papers  (2026-09-10T14:20:08Z)
• Verdict Instability of OOD Scores under Reference Resampling [4.260015792110957]
  Post-hoc out-of-distribution detectors are fitted on a finite reference set, so every score they produce is an estimate.<n>We measure verdict instability by resampling the reference set and recording the bootstrap standard deviation of the score.<n>We give a rule that predicts this sign for any score from a single label-free correlation, and abstention driven by a wrong-signed score turns out worse than abstention at random on every dataset we test.
  arXiv  Detail & Related papers  (2026-09-01T04:12:15Z)
• EDGE: a closed-form directed test for the calibration of probabilistic binary classifiers [0.0]
  We propose EDGE, a calibration test for the canonical probabilistic classifier, logistic regression.<n> EDGE reads the same binned predicted-observed table a reliability diagram plots, and projects its standardized bin residuals onto a small pre-specified basis of smooth calibration-distortion.<n>Across link and feature misspecification the pre-specified default led or tied every rival binned test on the fitted index in 19 of 22 detectable scenarios, and stayed computable where the refit-based Stukel score test separates in 20% to 28% of samples.
  arXiv  Detail & Related papers  (2026-08-20T19:06:05Z)
• Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees [1.0742675209112622]
  finite optimization can exploit evaluator errors rather than improve response quality.<n>We prove that common-mode error is not identifiable from internal judge scores alone.<n>We propose a bounded two-anchor Bernstein certificate for finite-search error and regret.
  arXiv  Detail & Related papers  (2026-08-08T08:28:55Z)
• The C-index illusion: discrimination without calibration in published survival models [0.0]
  We reproduce three published survival-ML models across three structurally distinct domains.<n>We test five pre-registered hypotheses under a Holm-corrected family-wise error rate.
  arXiv  Detail & Related papers  (2026-07-21T19:13:54Z)
• A prior-free blind detection of information leakage from model predictions [0.0]
  We give a decision-theoretic framework in which leakage diagnostics are functionals of the predicted-risk/outcome law.<n>A recalibrated leak matching an honest model's calibration and discrimination is indistinguishable from honest performance by emphany function of the predictions.<n>A near-deterministic subgroup produces a sustained unit-purity head that no legitimate predictor of a non-deterministic outcome can manufacture.
  arXiv  Detail & Related papers  (2026-06-09T05:13:03Z)
• Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection [56.362776482614976]
  Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection.<n>We show that this protocol can become ill-posed when the held-out anomaly class overlaps the normal mixture in representation space.<n>We introduce a simple training-free diagnostic, neighborhood class leakage, and show that it predicts score-direction instability across Fashion-MNIST, CIFAR-10, and Imagenette.
  arXiv  Detail & Related papers  (2026-05-23T06:27:42Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.