Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
- URL: http://arxiv.org/abs/2605.06283v1
- Date: Thu, 07 May 2026 13:55:11 GMT
- Title: Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
- Abstract summary: Autoraters are increasingly used for evaluation and automated moderation content.<n>There is limited statistical analysis of how modifications in a rubric presented to both humans and autoraters affect their score agreement.
- Score: 50.34437999083224
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Autoraters, also referred to as LLM-as-judges, are increasingly used for evaluation and automated content moderation. However, there is limited statistical analysis of how modifications in a rubric presented to both humans and autoraters affect their score agreement. Rubrics that ask for an overall or \emph{holistic} judgment - for example, rating the ``quality'' of an essay - may be inconsistently interpreted due to the complexity or subjectivity of the criteria. Conversely, rubrics can ask for \emph{analytic} judgments, which decompose assessment criteria - for example, ``quality'' into ``fluency'' and ``organization''. While these rubrics can be edited to improve the individual accuracy of both human and automated scoring, this approach may result in disagreement between the two scores, or with the associated holistic judgment. Designing and deploying reliable autoraters requires understanding not just the relationship between human and autorater annotations but how that relationship changes as holistic or analytic judgments are elicited. The results indicate that rubric edits providing representative examples and additional context, and reducing positional bias in the rubric increased human-autorater agreement, while higher rubric complexity and conservative aggregation methods tended to decrease it. The findings from the automatic essay scoring and instruction-following evaluation domains suggest that practitioners should carefully analyze domain- and rubric-specific performance to move towards higher human-autorater agreement.
Related papers
- Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation [20.285213577666315]
We compare a single-judge baseline against a consensus-based MAD protocol on subjective evaluation tasks.<n>Our ablations demonstrate that this performance drop stems primarily from asymmetric role prompting rather than the interaction itself.<n>These findings demonstrate that multi-agent consensus can enforce artificial agreement at the expense of true human alignment.
arXiv Detail & Related papers (2026-08-31T07:28:55Z) - When AI reviews science: Can we trust the referee? [73.47745294608072]
We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
arXiv Detail & Related papers (2026-04-26T08:03:32Z) - Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
arXiv Detail & Related papers (2026-04-21T14:21:15Z) - Artificial-Intelligence Grading Assistance for Handwritten Components of a Calculus Exam [41.99844472131922]
In a large first-year exam, students' handwritten work was graded by GPT-5 against the same rubric used by teaching assistants (TAs)<n>We calibrated a human-in-the-loop filter that combines a partial-credit threshold with an Item Response Theory (2PL) risk measure based on the deviation between the AI score and the model-expected score for each student-item.<n>Unfiltered AI-TA agreement was moderate, adequate for low-stakes feedback but not for high-stakes use.
arXiv Detail & Related papers (2025-10-04T15:07:06Z) - Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework [55.078301794183496]
We focus on a core reviewing skill that underpins high-quality peer review: detecting faulty research logic.<n>This involves evaluating the internal consistency between a paper's results, interpretations, and claims.<n>We present a fully automated counterfactual evaluation framework that isolates and tests this skill under controlled conditions.
arXiv Detail & Related papers (2025-08-29T08:48:00Z) - Analytical and Empirical Study of Herding Effects in Recommendation Systems [72.6693986712978]
We study how to manage product ratings via rating aggregation rules and shortlisted representative reviews.
We show that proper recency aware rating aggregation rules can improve the speed of convergence in Amazon and TripAdvisor.
arXiv Detail & Related papers (2024-08-20T14:29:23Z) - Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
arXiv Detail & Related papers (2023-05-24T06:19:14Z) - Rater Equivalence: Evaluating Classifiers in Human Judgment Settings [11.529701822081394]
We introduce a framework for evaluating automated classifiers based solely on human judgments.<n>Our framework uses human-generated labels both to construct benchmark panels and to evaluate performance.<n>Using case studies and formal analysis, we demonstrate how this framework can inform the evaluation and deployment of AI systems.
arXiv Detail & Related papers (2021-06-02T16:07:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.