Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices

• URL: http://arxiv.org/abs/2511.21860v1
• Date: Wed, 26 Nov 2025 19:35:48 GMT
• Title: Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices
• Abstract summary: We present the Consistency-Rebalanced Accuracy (CoRA) metric, improving the reliability of Large Language Model (LLM) scores computed on multiple choice (MC) benchmarks.<n>Our metric explores the response consistency of the LLMs, taking advantage of synthetically-generated questions with altered answer choices.
• Score: 0.013048920509133805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work we present the Consistency-Rebalanced Accuracy (CoRA) metric, improving the reliability of Large Language Model (LLM) scores computed on multiple choice (MC) benchmarks. Our metric explores the response consistency of the LLMs, taking advantage of synthetically-generated questions with altered answer choices. With two intermediate scores, i.e. Bare-Minimum-Consistency Accuracy (BMCA) and Consistency Index (CI), CoRA is computed by adjusting the multiple-choice question answering (MCQA) scores to better reflect the level of consistency of the LLM. We present evaluations in different benchmarks using diverse LLMs, and not only demonstrate that LLMs can present low response consistency even when they present high MCQA scores, but also that CoRA can successfully scale down the scores of inconsistent models.

Related papers

• Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation [52.58737865652009]
  Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy)<n>In educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward.<n>We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy.
  arXiv  Detail & Related papers  (2026-08-30T16:35:22Z)
• Boosting Self-Consistency with Ranking [56.38798757709555]
  Self-consistency improves large language models by sampling multiple reasoning paths and selecting the most frequent answer.<n>We address this limitation with Ranking-Improved Self-Consistency (RISC), which reformulates answer selection in self-consistency as a ranking problem.
  arXiv  Detail & Related papers  (2026-06-03T16:12:30Z)
• RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty [102.02839046225468]
  RankLLM is a novel framework designed to quantify both question difficulty and model competency.<n>We evaluate 30 models on 35,550 questions across multiple domains.
  arXiv  Detail & Related papers  (2026-02-12T21:28:46Z)
• JudgeBoard: Benchmarking and Enhancing Small Language Models for Reasoning Evaluation [13.831735556002426]
  Small language models (SLMs) have shown promise on various reasoning tasks.<n>Their ability to judge the correctness of answers remains unclear compared to large language models (LLMs)
  arXiv  Detail & Related papers  (2025-11-20T01:14:39Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• On Robustness and Reliability of Benchmark-Based Evaluation of LLMs [6.121856629864516]
  Large Language Models (LLMs) effectiveness is usually evaluated by means of benchmarks such as MMLU, ARC-C, or HellaSwag.<n>Real-world applications involve linguistic variability, requiring models to maintain their effectiveness across diverse rewordings of the same question or query.<n>We systematically assess the robustness of LLMs to paraphrased benchmark questions and investigate whether benchmark-based evaluations provide a reliable measure of model capabilities.
  arXiv  Detail & Related papers  (2025-09-04T08:43:27Z)
• Metric assessment protocol in the context of answer fluctuation on MCQ tasks [4.453107218424601]
  Using multiple-choice questions (MCQs) has become a standard for assessing LLM capabilities efficiently.<n>Previous research has not conducted a thorough assessment of them.<n>We suggest a metric assessment protocol in which evaluation methodologies are analyzed through their connection with fluctuation rates.
  arXiv  Detail & Related papers  (2025-07-21T13:01:46Z)
• CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks [63.562924932512765]
  Large Language Models (LLMs) have advanced the state-of-the-art in various coding tasks.<n>LLMs can also serve as judges, assessing and comparing the quality of responses generated by other models.
  arXiv  Detail & Related papers  (2025-07-14T17:56:29Z)
• Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering [78.89231943329885]
  Multiple-Choice Question Answering (MCQA) is widely used to evaluate Large Language Models (LLMs)<n>We show that multiple factors can significantly impact the reported performance of LLMs.<n>We analyze whether existing answer extraction methods are aligned with human judgment.
  arXiv  Detail & Related papers  (2025-03-19T08:45:03Z)
• Grade Score: Quantifying LLM Performance in Option Selection [0.0]
  "Grade Score" is a novel metric designed to evaluate the consistency and fairness of Large Language Models (LLMs) The Grade Score combines Entropy, which measures order bias, and Mode Frequency, which assesses choice stability. The study explores techniques such as prompt engineering and option sampling strategies to optimize the Grade Score.
  arXiv  Detail & Related papers  (2024-06-17T19:29:39Z)
• LLMs May Perform MCQA by Selecting the Least Incorrect Option [29.202758753639078]
  Large Language Models (LLMs) have markedly enhanced performance across a variety of tasks.<n>The adoption of Multiple Choice Question Answering (MCQA) as a benchmark for assessing LLMs has gained considerable traction.<n>However, concerns regarding the robustness of this evaluative method persist.
  arXiv  Detail & Related papers  (2024-02-02T12:07:00Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.