Should I Be Polite to My LLM Relevance Judge? Tone as a Severity Operating-Point Shift

• URL: http://arxiv.org/abs/2609.09703v1
• Date: Wed, 09 Sep 2026 04:40:14 GMT
• Title: Should I Be Polite to My LLM Relevance Judge? Tone as a Severity Operating-Point Shift
• Abstract summary: Tone affects calibration-based agreement more than ranking outcomes.<n>Where tone changes agreement, the results are more consistent with a shift in the judge's severity operating point.
• Score: 7.837842601637295
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly used as relevance judges, yet their labels can shift with prompt surface form. We study one such feature -- tone -- on 3,498 TREC DL19/DL20 query-passage pairs, across eight judge models, five classifier-calibrated politeness levels, and three paraphrases per level. Effects are strongly model-dependent: one judge shows a structured U-shaped response, whereas most show only small changes. Where tone changes agreement, the results are more consistent with a shift in the judge's severity operating point -- its overall scoring leniency -- than with improved judgment. Agreement rises or falls as this shift moves the judge toward or away from human annotators' strictness. A query-disjoint cross-fit retains the expected association (Spearman $ρ= -0.683$; exact model-block permutation $p = 0.019$). Tone affects calibration-based agreement more than ranking outcomes: across 32 model-tone contrasts, the largest absolute mean change in NDCG@10 is 0.011, although Kendall's $τ$ as low as 0.743 shows that reordering is reduced, not absent. The account reconciles prior contradictory findings and identifies prompt tone as a potential validity threat when absolute relevance labels matter.

Related papers

• Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit [15.81584544074437]
  Audits of LLM judges certify a bias by contrasting matched conditions.<n>We show that this endpoint is not identified on the scale that reports it.<n>We derive the mechanism in closed form and show that its contribution is measurable from an audit's own ratings.
  arXiv  Detail & Related papers  (2026-08-27T16:10:16Z)
• Beyond Accuracy: A Dual-Judge Evaluation Protocol for Vision-Language Models in Legally Grounded Tasks [1.2555090617748867]
  We contribute a dual-judge protocol that pairs a standard 0-10 quality judge with a strict binary semantic-equivalence judge against a human-curated reference.<n>We study a controlled, visually grounded regulatory task, whose meaning is a codified question with a known reference for every input.
  arXiv  Detail & Related papers  (2026-08-25T08:49:46Z)
• Sharding Prevents LLM Oversight Failures and Adversarial Exploitation [45.380649793616705]
  Sharding partitions requirements into smaller groups, assigns each group to a separate call, and aggregates the verdicts.<n>We find that a sharded weaker judge can outperform a more capable holistic judge and match that judge even when the latter receives the panel's full budget.
  arXiv  Detail & Related papers  (2026-08-05T18:21:21Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain [0.0]
  Hate speech annotation pipelines collapse annotator disagreement into majority vote labels before training.<n>We show that this aggregation is not neutral.<n>Three downstream interventions of increasing sophistication all fail to recover boundary accuracy.
  arXiv  Detail & Related papers  (2026-06-27T06:56:07Z)
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels [0.0]
  LLM-as-a-judge panels aggregate votes from multiple models.<n>We quantify how far their reliability falls short of the independent-voting ideal.
  arXiv  Detail & Related papers  (2026-05-28T11:48:17Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior [0.0]
  We find that judges are consistent, but not with each other; they are consistent with themselves.<n>Across 3,240 evaluations, inter-judge agreement is near-zero.<n>Averaging judges' scores produces a synthetic verdict that corresponds to no judge's actual values.
  arXiv  Detail & Related papers  (2026-01-08T17:02:22Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)
• The price of unfairness in linear bandits with biased feedback [62.25313751895011]
  We study the problem of sequential decision making with biased linear bandit feedback. We show that the worst case regret is higher than the dT 1/2 log(T) regret rate obtained under unbiased feedback. Interestingly, the gap-dependent rates reveal the existence of non-trivial instances where the problem is no more difficult than its unbiased counterpart.
  arXiv  Detail & Related papers  (2022-03-18T08:03:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.