A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation

• URL: http://arxiv.org/abs/2608.24419v1
• Date: Tue, 25 Aug 2026 11:32:54 GMT
• Title: A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation
• Abstract summary: We formalize construct validity for an evaluator as a two-dimensional profile.<n>We show that S and R are independent and that no scalar summary preserves all relevant comparisons.<n>These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated.
• Score: 14.867869624585886
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-as-a-judge evaluation is usually assessed by agreement and robustness to surface perturbations, but reliability does not establish construct validity. We formalize construct validity for an evaluator as a two-dimensional profile: invariance S, the probability that a verdict is unchanged under construct-preserving edits, and construct sensitivity R, the probability that it changes under minimal construct-changing edits. We show that S and R are independent and that no scalar summary preserves all relevant comparisons. We measure the profile across 7 judges and 4 domains using 7 construct-changing intervention types and 5 register-only controls, with intervention direction determined by human annotators and generation, verification, and judging assigned to disjoint model families. At matched invariance S >= 0.90, judges average S = 0.945 but R = 0.319. Sensitivity also differs between scope and strength edits: R_scope = 0.383 versus R_strength = 0.262, a +0.121 gap with the same sign for all 7 judges. We further audit five public label sets and find that surface-only predictors reproduce 55%-67% of labels in paired mode, including 67.4% of MT-Bench human votes. These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated, motivating joint reporting of invariance and sensitivity and auditing the validation set itself.

Related papers

• Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging [41.62710134204231]
  We study the failure mode through psychological depth in short stories.<n>A judge's correlation with human ratings on its development set may not guarantee valid measurement when outputs are closely matched and human preferences are subjective.<n>These results suggest that development-set performance is insufficient evidence for deployment validity on a shifted distribution.
  arXiv  Detail & Related papers  (2026-09-12T07:34:54Z)
• Counterfactual Fairness Audits of Multi-Step Clinical LLM Agents Require a Measured Per-Action Instability Floor [3.0704370207786265]
  We show that a counterfactual fairness estimate reported without a per-action floor beside it cannot be read as evidence of disparity.<n>The measurements were taken with FairMedAgent, an evaluation harness for disparity in the actions of clinical adjudication agents.
  arXiv  Detail & Related papers  (2026-09-02T23:38:17Z)
• Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation [0.1269104766024433]
  We study 99,952 public, rubric-conditioned examples.<n>Supplying the correct rubric improves locked-test accuracy by 2.11 points over a response-only control.<n>On RewardBench 2, learned correctness heads route examples through a 0.6B-4B-8B cascade without changing any reward score.
  arXiv  Detail & Related papers  (2026-07-30T10:29:22Z)
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks [47.027290803102666]
  We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation.<n>A pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark.
  arXiv  Detail & Related papers  (2026-06-02T13:41:43Z)
• Let the Results Speak: A Replication-First Paradigm for LLM Behavioral Benchmarking [22.825786049667602]
  We propose a replication-first paradigm to anchor validity to a single human-rater consensus.<n>We certify the instrument via four properties -- reliability across K runs, cross-instrument replication across architecturally distinct judges, historical-footprint calibration via judges from earlier training cohorts, and pre-registered prediction.<n>We test it on emotional accompaniment by letting the self-evolve data-driven across: the dimensions are not pre-stipulated and the procedure stabilizes to a 9-dimension set.
  arXiv  Detail & Related papers  (2026-05-27T03:41:11Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models [0.0]
  Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter.<n>We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks.<n>We find three broad results. First, evaluation design can materially change the conclusion.<n>Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant.
  arXiv  Detail & Related papers  (2026-05-03T20:05:08Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• SpatialBench-UC: Uncertainty-Aware Evaluation of Spatial Prompt Following in Text-to-Image Generation [0.0]
  SpatialBench-UC is a small, reproducible benchmark for pairwise spatial relations.<n>We release a benchmark package, versioned prompts, pinned configs, per-sample checker outputs, and report tables.<n>We evaluate three baselines, Stable Diffusion 1.5, SD 1.5 BoxDiff, and SD 1.4 GLIGEN.
  arXiv  Detail & Related papers  (2026-01-19T23:37:10Z)
• Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior [0.0]
  We find that judges are consistent, but not with each other; they are consistent with themselves.<n>Across 3,240 evaluations, inter-judge agreement is near-zero.<n>Averaging judges' scores produces a synthetic verdict that corresponds to no judge's actual values.
  arXiv  Detail & Related papers  (2026-01-08T17:02:22Z)
• ORCA: Open-ended Response Correctness Assessment for Audio Question Answering [41.72231074041232]
  We present ORCA, a framework that models the variability in human judgments using Beta distributions to predict both expected correctness and uncertainty.<n>We collected 11,721 annotations across 3,580 question-answer pairs from 15 LALMs on two audio QA benchmarks, achieving inter-annotator agreement of 0.82 (Krippendorff's alpha)
  arXiv  Detail & Related papers  (2025-11-28T14:41:48Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.