PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

• URL: http://arxiv.org/abs/2605.30803v1
• Date: Fri, 29 May 2026 03:45:11 GMT
• Title: PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges
• Abstract summary: A vague rubric asking for a response to be helpful and factual'' can reward polished answers that invent facts or violate user intent.<n>We introduce PReMISE, a framework that audits any rubric set under LLM-judge use along four axes: structural adequacy, reliability, preference fit, and adversarial robustness.<n>We contribute two audit-targeted repair operations: preference-rank selection raises judge accuracy on paired responses from $65.0%$ to $68.6%$, competitive with the strongest rubric-discovery baselines, and leading on two of three judges in our cross-judge sweep.
• Score: 33.739528721872844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM judges are increasingly used to evaluate open-ended responses, but their scores depend strongly on the rubrics that condition them. A vague rubric asking for a response to be ``helpful and factual'' can reward polished answers that invent facts or violate user intent. We treat reusable rubrics as measurement specifications: changing the rubric changes the response quality measurement induced by a fixed judge. We introduce PReMISE, a framework that, given pairwise human-preference data, (i) discovers a policy-level rubric set, and (ii) audits any rubric set under LLM-judge use along four axes: structural adequacy, reliability, preference fit, and adversarial robustness. Across rubric sources no raw source is simultaneously reliable, preference-predictive, and adversarially robust; and high inter-rater agreement does not imply low exploitability. PReMISE is the only rubric source to score non-trivially on applicability, specificity, and effective dimensionality simultaneously. We contribute two audit-targeted repair operations: preference-rank selection raises judge accuracy on paired responses from $65.0\%$ to $68.6\%$, competitive with the strongest rubric-discovery baselines and leading on two of three judges in our cross-judge sweep; reliability-constrained refinement reduces the rate at which exploit responses receive high scores from $46.4\%$ to $36.0\%$ with little change in inter-judge agreement ($α{=}.531\to.519$).

Related papers

• Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration [0.5294698352039444]
  We study absolute scoring tasks, which mirror more realistic use cases.<n>We show that a model's task accuracy strongly predicts its judging accuracy.<n>More capable examinee models consistently receive more lenient judgments from all judges.
  arXiv  Detail & Related papers  (2026-09-10T00:25:39Z)
• Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees [16.73004625486562]
  Using LLMs as judges has become standard practice for evaluating model outputs at scale.<n>In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation.<n>We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level.
  arXiv  Detail & Related papers  (2026-08-18T16:42:02Z)
• CalibratedRubric: Task-Adaptive Rubric Banks for Open-Ended LLM Evaluation [11.869490719319577]
  CalibratedRubric is a task-adaptive framework that combines type-specific scoring, Bayesian rubric-measurability filtering, and item response theory (IRT)-based bank assembly.<n>Across financial, healthcare, general, and legal benchmarks, measurability filtering improves human-gold agreement.<n>IRT-based greedy selection improves cross-fitted rank fidelity over random selection.
  arXiv  Detail & Related papers  (2026-07-31T10:21:56Z)
• Instance-Optimal Estimation with Multiple LLM Judges on a Budget [84.31744861038106]
  We formalize this question as *budgeted heteroskedastic multi-judge estimation*.<n>Given $K$ prompt-response pairs, $J$ judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an $ell_p$-error.<n>We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget.
  arXiv  Detail & Related papers  (2026-05-22T08:26:08Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Judge Reliability Harness: Stress Testing the Reliability of LLM Judges [1.1699027359021665]
  Judge Reliability Harness is an open source library for constructing validation suites that test the reliability of LLM judges.<n>We evaluate four state-of-the-art judges across four benchmarks spanning safety, persuasion, misuse, and agentic behavior.
  arXiv  Detail & Related papers  (2026-03-05T17:27:07Z)
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning [0.6138671548064355]
  Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning.<n>We introduce C2-Faith, a benchmark that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and coverage.<n>We evaluate three frontier judges under three tasks: binary causal detection, causal step localization, and coverage scoring.
  arXiv  Detail & Related papers  (2026-03-05T13:36:47Z)
• SCOPE: Selective Conformal Optimized Pairwise LLM Judging [16.73004625486562]
  Large language models (LLMs) are increasingly used as judges to replace costly human preference labels in pairwise evaluation.<n>Despite their practicality, LLM judges remain prone to miscalibration and systematic biases.<n>This paper proposes SCOPE, a framework for selective pairwise judging with finite-sample statistical guarantees.
  arXiv  Detail & Related papers  (2026-02-13T17:10:43Z)
• Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems [32.83708359216193]
  Large Language Models (LLMs) are increasingly being used to autonomously evaluate the quality of content in communication systems.<n>This paper systematically investigates judgment biases in two LLM-as-a-judge models under the point-wise scoring setting.<n>We propose four potential mitigation strategies to ensure fair and reliable AI judging in practical communication scenarios.
  arXiv  Detail & Related papers  (2025-10-14T12:52:29Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• DAFE: LLM-Based Evaluation Through Dynamic Arbitration for Free-Form Question-Answering [12.879551933541345]
  We propose the Dynamic Arbitration Framework for Evaluation (DAFE) to evaluate large language models.<n>DAFE employs two primary LLM-as-judges and engages a third arbitrator only in cases of disagreements.<n>We show DAFE's ability to provide consistent, scalable, and resource-efficient assessments.
  arXiv  Detail & Related papers  (2025-03-11T15:29:55Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.