Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation

• URL: http://arxiv.org/abs/2608.05353v2
• Date: Tue, 11 Aug 2026 02:22:28 GMT
• Title: Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
• Abstract summary: This workflow assumes that the intermediate record preserves the information needed for a later verdict.<n>For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative.<n>We compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5.
• Score: 0.6768558752130311
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM judges are often asked to extract criteria and evidence before choosing between candidate answers. This workflow assumes that the intermediate record preserves the information needed for a later verdict. For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative: persist the evidence in one call and make it the exclusive input to the next. Across 24,000 judgments over HelpSteer3, FeedbackQA, and CoVal, we compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5. Evidence locking reduces agreement with released human preferences by 4 to 6 percentage points and increases answer-order inconsistency by 8 to 10 points relative to structured one-call judging. Pointwise locking is also harmful, while structured evidence elicitation remains close to standard judging. The result holds for both judges and all three datasets. Persisted evidence can support auditability, but it should not replace the source answers at decision time.

Related papers

• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees [16.73004625486562]
  Using LLMs as judges has become standard practice for evaluating model outputs at scale.<n>In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation.<n>We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level.
  arXiv  Detail & Related papers  (2026-08-18T16:42:02Z)
• SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification [71.9783246097687]
  SEER hides candidate relations during pair localization, constructs a query-specific view with explicit subject/object roles, and retains the full image and sparse box geometry as complementary evidence.<n>For relation-choice protocols with exact inverse support, an optional refinement swaps the entity roles and changes the forward decision only when exactly one visual state obeys the corresponding inverse relation.<n>The unchanged protocol yields +4.35 to +11.79 on all 2,434 filtered EmbSpatial pair-relation questions across three models.
  arXiv  Detail & Related papers  (2026-08-04T13:16:15Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety [0.0]
  We stress-test four models by deleting half the final user turn in HealthBench conversations, grading responses with a four-provider LLM-judge panel and a blinded clinician-anchored reference.<n>Two evaluator-facing results are robust. First, judge choice materially changes apparent safety.<n>Second, LLM judges are more permissive than clinicians on a blinded 50-item subsample.
  arXiv  Detail & Related papers  (2026-07-21T08:05:35Z)
• When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability [0.0]
  An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.<n>We compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs.<n>The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax releases do not.
  arXiv  Detail & Related papers  (2026-07-09T14:31:05Z)
• How Sensitive Are Safety Benchmarks to Judge Configuration Choices? [5.94231111588812]
  Safety benchmarks such as HarmBench rely on judges to classify model responses as harmful or safe.<n>We show that the combination of judge model and judge prompt is typically treated as a fixed implementation detail.<n>We find that prompt wording alone, holding the judge model fixed, shifts measured harmful-response rates by up to 24.2 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T05:59:59Z)
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning [0.6138671548064355]
  Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning.<n>We introduce C2-Faith, a benchmark that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and coverage.<n>We evaluate three frontier judges under three tasks: binary causal detection, causal step localization, and coverage scoring.
  arXiv  Detail & Related papers  (2026-03-05T13:36:47Z)
• The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluation [17.386684382460242]
  Large language models (LLMs) are increasingly used to evaluate system outputs in tasks such as reasoning, question answering, and creative writing.<n>We test this ideal via controlled cue perturbations-synthetic metadata labels injected into evaluation prompts for six judge models.<n>We study six cue families: source, temporal, age, gender, ethnicity, and educational status.
  arXiv  Detail & Related papers  (2026-02-08T14:45:23Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.