Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation
Abstract Overview
This paper examines how multiple-choice question answering benchmarks change when evaluated with alternative scoring schemes rather than standard number-right accuracy. Drawing on educational testing, the authors implement six schemes that target abilities beyond simple correctness, including abstention, distractor elimination, confidence calibration, and self-correction. They evaluate 31 LLMs on ARC, MMLU, and SuperGPQA using prompt and metric changes only. The analysis focuses on whether these schemes alter model rankings, better align with human preferences, and reveal behavioral differences that accuracy alone obscures.
Novelty
The paper’s main novelty is to treat scoring scheme design itself as a core evaluation choice for NLP MCQA, adapting six education-inspired schemes into a unified LLM benchmarking framework. Rather than proposing a new dataset or model, it demonstrates that alternative response modes and grading rules surface distinct capabilities without altering underlying task inputs.
Results
Elimination-based scoring, personal point allocation, and answer-until-correct shift model rankings more than a prompt-rephrasing baseline, confirming they capture distinct abilities from number-right accuracy. Answer-until-correct yields the strongest correlation with user preferences on LLM Arena. Furthermore, cross-scheme evaluations uncover behavioral traits, such as stronger GPT models rarely abstaining and adapting quickly to feedback, whereas weaker open-weight models frequently abstain or hesitate to eliminate choices.
Key Points
- The study adapts six education-grounded scoring schemes to test abstention, eliminative reasoning, confidence calibration, and self-correction in LLMs on MCQA benchmarks.
- Alternative scoring rules materially reorder model rankings compared to standard accuracy, demonstrating that benchmark results strongly depend on the targeted behaviors rewarded.
- Answer-until-correct correlates most strongly with human preferences in LLM Arena and exposes model-specific behavioral tendencies that number-right scoring obscures.