Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees

• URL: http://arxiv.org/abs/2608.08002v1
• Date: Sat, 08 Aug 2026 08:28:55 GMT
• Title: Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees
• Abstract summary: finite optimization can exploit evaluator errors rather than improve response quality.<n>We prove that common-mode error is not identifiable from internal judge scores alone.<n>We propose a bounded two-anchor Bernstein certificate for finite-search error and regret.
• Score: 1.0742675209112622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language-model judges and reward models enable scalable supervision, but finite optimization can exploit evaluator errors rather than improve response quality. We characterize this failure through the covariance geometry of evaluator ensembles. For calibrated judges, the ensemble mean retains common-mode error along the all-ones direction, whereas cross-judge disagreement captures only orthogonal error. Consequently, disagreement can be high despite robust aggregation, or low while shared response-dependent errors persist. We prove that common-mode error is not identifiable from internal judge scores alone. Under a joint sub-Gaussian model, we bound best-of-K selection overstatement and target-quality regret, extending the guarantees to predictably adaptive search under conditional calibration. The resulting search terms scale as the square root of log K and are asymptotically tight for Gaussian projected errors. We further show that noisy quality proxies introduce artificial rank-one covariance without changing disagreement, and propose a bounded two-anchor Bernstein certificate for finite-search error and regret. Fixed-seed Gaussian stress tests over 120 (J, rho, K) configurations and real-model audits validate the theory while revealing the limits of disagreement-based diagnostics under increasing search pressure.

Related papers

• A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Model [0.0]
  An external reference set is used to decompose an LLM-judge panel's error into a quality signal and a shared common-mode error.<n>We study when that assumption can be dropped and replaced by an estimate.<n>A clean-anchor estimator reports a contaminated companion anchor as fully clean once its trusted anchor is itself contaminated.
  arXiv  Detail & Related papers  (2026-09-08T14:49:13Z)
• Theory of approximate quantum error correction and the error-set model [53.73800998389749]
  We develop a theory of approximate quantum error correction based on the error-set model.<n>Code correcting a given error set automatically protects against every channel whose Kraus operators lie in their linear span.<n>This linearity gives rise to code distance, the equivalence between erasures and general errors, and a theory ofally good codes.
  arXiv  Detail & Related papers  (2026-07-25T02:15:33Z)
• Precise sample covariance spectral norm error -- an RDT view [2.538209532048867]
  We develop a generic framework based on Random Duality Theory (RDT)<n>We establish complementary lower bounds by introducing a novel bilinear-quadratic RDT lower-bounding mechanism.<n>Our theoretical results are supplemented with numerical evaluations and simulations.
  arXiv  Detail & Related papers  (2026-07-16T01:16:31Z)
• Denoised Conformal Alignment for Reliable Selection of Conditional Average Treatment Effect Predictions [10.945117945979566]
  We study reliable selection for black-box CATE predictors.<n>We construct doubly robust proxy errors from pseudo-outcomes.<n>We propose Denoised Conformal Alignment.
  arXiv  Detail & Related papers  (2026-07-03T10:00:28Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• What Does It Take to Build a Performant Selective Classifier? [30.90225954725644]
  Bayes noise, approximation error, ranking error, statistical noise, and implementation- or shift-induced slack are studied.<n>We validate our decomposition on synthetic two-moons data and on real-world vision and language benchmarks.<n>Our results confirm that (i) Bayes noise and limited model capacity can account for substantial gaps, (ii) only richer, feature-aware calibrators meaningfully improve score ordering, and (iii) data shift introduces a separate slack that demands distributionally robust training.
  arXiv  Detail & Related papers  (2025-10-23T05:48:40Z)
• Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal [31.458406135473805]
  We present UniCR, a unified framework that turns heterogeneous uncertainty evidence into a calibrated probability of correctness.<n>UniCR learns a lightweight calibration head with temperature scaling and proper scoring.<n>Experiments on short-form QA, code generation with execution tests, and retrieval-augmented long-form QA show consistent improvements in calibration metrics.
  arXiv  Detail & Related papers  (2025-09-01T13:14:58Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Latent Veracity Inference for Identifying Errors in Stepwise Reasoning [78.29317733206643]
  We introduce Veracity Search (VS), a discrete search algorithm over veracity assignments.<n>It performs otherwise intractable inference in the posterior distribution over latent veracity values.<n>It generalizes VS, enabling accurate zero-shot veracity inference in novel contexts.
  arXiv  Detail & Related papers  (2025-05-17T04:16:36Z)
• Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum [56.37522020675243]
  We provide the first proof of convergence for normalized error feedback algorithms across a wide range of machine learning problems. We show that due to their larger allowable stepsizes, our new normalized error feedback algorithms outperform their non-normalized counterparts on various tasks.
  arXiv  Detail & Related papers  (2024-10-22T10:19:27Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity [49.66890309455787]
  We introduce the expected co-coercivity condition, explain its benefits, and provide the first last-iterate convergence guarantees of SGDA and SCO. We prove linear convergence of both methods to a neighborhood of the solution when they use constant step-size. Our convergence guarantees hold under the arbitrary sampling paradigm, and we give insights into the complexity of minibatching.
  arXiv  Detail & Related papers  (2021-06-30T18:32:46Z)
• Sample Complexity of Uniform Convergence for Multicalibration [43.10452387619829]
  We address the multicalibration error and decouple it from the prediction error. Our work gives sample complexity bounds for uniform convergence guarantees of multicalibration error.
  arXiv  Detail & Related papers  (2020-05-04T18:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.