Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit

• URL: http://arxiv.org/abs/2608.27309v2
• Date: Wed, 02 Sep 2026 05:16:14 GMT
• Title: Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit
• Abstract summary: Audits of LLM judges certify a bias by contrasting matched conditions.<n>We show that this endpoint is not identified on the scale that reports it.<n>We derive the mechanism in closed form and show that its contribution is measurable from an audit's own ratings.
• Score: 15.81584544074437
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Audits of LLM judges certify a bias by contrasting matched conditions, and the strongest designs difference twice: a within-item contrast between two candidate responses, differenced again across a manipulated attribute, read off a bounded rating scale. We show that this endpoint is not identified on the scale that reports it. Each term of the double difference is censored by its own share, so the observed statistic confounds differential preference with differential attenuation: a severity shift common to both responses manufactures an interaction whenever the two censor it unequally, as unequal distances from the bounds make them, exactly where good stimuli place them. We exhibit the failure inside a pre-registered audit of a frozen pedagogy judge, sealed before the first of its 990 calls. The registered primary endpoint, the effect of a stated learner profile on the judge's scaffolding preference, is null: $+0.085$ points (95\% BCa $[-0.167, +0.353]$, $p = 0.684$). The audit's one nominally significant interaction, $+0.378$ ($p = 0.002$), is not identified as preference: a construction containing zero differential preference reproduces 79 to 85\% of it from the observed severity shift and the scale floor alone. We derive the mechanism in closed form and show that its contribution is measurable from an audit's own ratings.

Related papers

• Unmatched Does Not Mean False: Incomplete Reference Sets Can Reverse Calibration Rankings in Open-Ended Theory-of-Mind Tracking [0.5735035463793009]
  Open-ended Theory-of-Mind (ToM) trackers emit valid beliefs absent from finite references.<n>A finite-reference-plus-matcher pipeline marks unmatched outputs false, creating proxy labels that can reverse proper-score model selection.
  arXiv  Detail & Related papers  (2026-08-26T11:36:31Z)
• A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation [14.867869624585886]
  We formalize construct validity for an evaluator as a two-dimensional profile.<n>We show that S and R are independent and that no scalar summary preserves all relevant comparisons.<n>These results show that high judge agreement can coexist with weak sensitivity to changes in the construct being evaluated.
  arXiv  Detail & Related papers  (2026-08-25T11:32:54Z)
• Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification [9.710464466895521]
  Quoting independent visual judgments creates large anchoring gaps under both self and peer framing.<n>We introduce panel-consensus verification, which cross-checks a quote against independently collected blind votes.
  arXiv  Detail & Related papers  (2026-08-08T04:55:45Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games [50.880420636090896]
  We build an auditable framework that maintains an external belief state over hidden roles in a 9-player Werewolf environment.<n>We report the effect as an association and treat its mechanism as unresolved.
  arXiv  Detail & Related papers  (2026-07-12T16:03:30Z)
• PerceptionRubrics: Calibrating Multimodal Evaluation to Human Perception [75.70273182182397]
  PerceptionRubrics is a rubric-based evaluation framework.<n>It addresses the gap between saturated benchmark scores and real-world brittleness.<n>It pairs 1,038 information-dense images with over 10,000 instance-specific rubrics.
  arXiv  Detail & Related papers  (2026-06-26T17:59:15Z)
• A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR [1.2958054117511815]
  Reinforcement learning from rewards improves reasoning even when the reward signal is spurious.<n> Practitioners commonly interpret naive = acc(TRUE) - acc(R) as the reward-design effect.<n>We prove this estimand is systematically biased.
  arXiv  Detail & Related papers  (2026-06-04T09:35:54Z)
• Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why [34.429649156970015]
  A survey of 24 recent LLM-as-judge papers finds metric choice entangled with the judgment scale, tie handling, invalid outputs, and abstention handling.<n>Most of the reported numbers are redundant: Pearson's $r$, Spearman's $$, Kendall's $_b$, the phi coefficient $$, and the Matthews Correlation Coefficient all reduce to a single number.
  arXiv  Detail & Related papers  (2026-05-25T07:31:44Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making [58.06306331390586]
  We introduce the notion of a margin complement, which measures how much a prediction score $S$ changes due to a thresholding operation. We show that under suitable causal assumptions, the influences of $X$ on the prediction score $S$ are equal to the influences of $X$ on the true outcome $Y$.
  arXiv  Detail & Related papers  (2024-05-24T11:22:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.