Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation

• URL: http://arxiv.org/abs/2608.30373v1
• Date: Mon, 31 Aug 2026 07:28:55 GMT
• Title: Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation
• Abstract summary: We compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks.<n>Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself.<n>These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment.
• Score: 20.285213577666315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. However, for subjective rubric-based scoring, inter-agent agreement does not guarantee alignment with human judgments. In this paper, we compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks and design three ablations to isolate the impact of role prompting, multi-round interaction, and explicit score sharing. Evaluations across six LLMs show that the single-judge baseline achieves the strongest human alignment on average across six judge models, whereas MAD shows degradation in human alignment on both tasks. Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself. Specifically, assigning a strict judge role introduces a systematic downward bias that the consensus process fails to correct. The central finding is that this bias reflects strict-stance dominance beyond averaging: the consensus score falls well beyond the arithmetic midpoint of the standalone strict and lenient conditions, rather than averaging them out. Removing role asymmetry (Symmetric MAD) largely recovers baseline performance, while masking peer scores widens inter-agent disagreement on average and worsens average human alignment. These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment, revealing a structural limitation in consensus-style, role-specialized MAD protocols for subjective scoring.

Related papers

• On the Adversarial Robustness of Multimodal LLM Judges [49.80009278197918]
  We introduce RobustMLLMJudge, the first framework for evaluating the adversarial robustness of general-purpose MLLMs when functioning as judges.<n>Using RobustMLLMJudge, we reveal that different MLLM judges are highly vulnerable to score-inflating adversarial attacks.<n>We propose MGSIA, a novel method that bypasses these constraints to enable more effective and transferable attacks on MLLM judges.
  arXiv  Detail & Related papers  (2026-06-14T05:30:20Z)
• MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring [6.092300706830162]
  We present MADRAG, a training-free framework for analytic essay scoring.<n>An Advocate identifies strengths, a Skeptic weaknesses, and a Judge aggregates their arguments into a final score.
  arXiv  Detail & Related papers  (2026-06-04T22:32:47Z)
• Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges [0.519554837386174]
  We study post-decision manipulability: the extent to which an evaluation outcome can be altered through subsequent conversation with the judge.<n>We find that LLM judges are highly stable under repeated and neutral reevaluation, yet become substantially reversible under targeted post-decision challenge.<n>These reversals have practical consequences: they can degrade agreement with human preferences, shift benchmark rankings, and produce harmful evaluation changes despite high self-reported confidence.
  arXiv  Detail & Related papers  (2026-06-03T19:37:23Z)
• Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling [35.945096782147864]
  MLLM judges tend to reward plausible narratives over perceptually correct answers when visual evidence conflicts with textual cues.<n>We introduce the Perceptually Perturbed Judgment dataset, which constructs minimally edited counterfactual responses.<n>We develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels.
  arXiv  Detail & Related papers  (2026-06-01T17:59:46Z)
• Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement [50.34437999083224]
  Autoraters are increasingly used for evaluation and automated moderation content.<n>There is limited statistical analysis of how modifications in a rubric presented to both humans and autoraters affect their score agreement.
  arXiv  Detail & Related papers  (2026-05-07T13:55:11Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents [41.829135334587626]
  MERRY is a semantically decoupled evaluation framework for assessing Multimodal Emotional and Role consistencies of Role-playing agents.<n>We transform the traditional subjective scoring approach into a novel bidirectional-evidence-finding task.<n>We conduct extensive evaluations based on MERRY.
  arXiv  Detail & Related papers  (2026-02-24T02:53:58Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge [23.497453639857852]
  We propose UDA, a framework that reduces inter-judge disagreement by dynamically adjusting the Elo rating system.<n>UDA operates in a fully unsupervised manner, guided solely by the objective of minimizing the dispersion among the Elo trajectories of all judges.<n>Experiments show that UDA significantly reduces the inter-judge rating standard deviation by up to 63.4% and improves the average correlation with human judgments by 24.7%.
  arXiv  Detail & Related papers  (2025-08-13T11:41:01Z)
• Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judge [70.89799989428367]
  We conduct a systematic analysis of four diverse bias types: position bias, verbosity bias, chain-of-thought bias, and bandwagon bias.<n>We evaluate these biases across two widely adopted multi-agent LLM-as-Judge frameworks: Multi-Agent-Debate and LLM-as-Meta-Judge.
  arXiv  Detail & Related papers  (2025-05-26T03:56:41Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.