Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks

• URL: http://arxiv.org/abs/2608.02966v1
• Date: Mon, 03 Aug 2026 23:59:39 GMT
• Title: Every Wrong Answer Counts: Option-Level Psychometrics for LLM Multiple-Choice Benchmarks
• Abstract summary: We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware framework that models the full distribution over answer choices.<n>We show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.
• Score: 19.555897987977445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most multiple-choice question (MCQ) benchmarks evaluate Large Language Models (LLMs) only by whether they select the correct answers. This binary scoring treats all incorrect responses alike, even though an LLM's preferences among incorrect options may contain systematic and useful information about its behavior and ability. We introduce the LLM Nominal Response Model (LLM-NRM), an option-aware psychometric framework that models the full distribution over answer choices to jointly estimate LLM ability and option-level item characteristics, while separating model-specific response calibration sharpness, positional preference, and difficulty-dependent fallback behavior. Across 189 LLMs and 31,554 items from 14 benchmarks, LLM-NRM predicts held-out LLM-item interactions more accurately than binary Item Response models and conventional nominal-response baselines, and its ability estimates achieve the strongest Spearman correlation of 0.920 with the external human-preference Arena.ai Elo leaderboard. Distractor identity contributes +101% additional Fisher Information per item beyond correctness, and incorrect responses alone recover full-information ability estimates with Spearman 0.943. The learned item parameters also enable efficient benchmarking, where 41 selected items preserve the full-bank ranking with Kendall's correlation 0.85, corresponding to a 770 times reduction. In conclusion, we show that incorrect answers carry distinct and useful measurement information rather than representing equivalent mistakes.

Related papers

• Check The Scoreboard: An Analysis of Scoring Schemes on Multiple-Choice Evaluation [52.58737865652009]
  Multiple-choice question answering (MCQA) benchmarks in NLP use number-right scoring (accuracy)<n>In educational testing, the scoring scheme, the combination of the response mode models follow and the rule for grading responses, is a key design choice that dictates which abilities to reward.<n>We examine how alternatives to number right change what MCQA measures with six education-inspired schemes that assess abilities beyond accuracy.
  arXiv  Detail & Related papers  (2026-08-30T16:35:22Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• Auditing LLM Benchmarks with Item Response Theory [1.1455131351238124]
  LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all.<n>We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples.<n>We trace these errors to mechanical labelings, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label.
  arXiv  Detail & Related papers  (2026-05-28T19:38:44Z)
• From Stochastic Answers to Verifiable Reasoning: Interpretable Decision-Making with LLM-Generated Code [0.0]
  Large language models (LLMs) are increasingly used for high-stakes decisionmaking.<n>Black-box models obscure their reasoning, while recent LLM-based rule systems rely on per-sample evaluation.<n>We propose reframing LLMs as code generators rather than per-instance evaluators.
  arXiv  Detail & Related papers  (2026-02-28T00:27:29Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models [67.62810111789338]
  Large Language Models exhibit a confidence distortion problem on multi-choice question-answering.<n>We propose Self-ensemble to solve this problem.<n> Experimental results on three LLMs and datasets demonstrate that Self-ensemble comprehensively addresses the confidence distortion problem.
  arXiv  Detail & Related papers  (2025-06-02T17:59:29Z)
• SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questions [17.12876663526222]
  Large language models (LLMs) are increasingly evaluated on single-answer multiple-choice tasks.<n>Many real-world problems require identifying all correct answers from a set of options.<n>We introduce SATA-BENCH, the first dedicated benchmark for evaluating LLMs on Select All That Apply questions.
  arXiv  Detail & Related papers  (2025-05-31T17:14:21Z)
• None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks [0.9831489366502301]
  We introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts.<n>Using this method, we evaluate state-of-the-art proprietary and open-source LLMs on two datasets available in English and Spanish.<n>Results show that all models experience remarkable accuracy drops, with an average loss of 57% on MMLU and 50% on UNED-Access 2024.
  arXiv  Detail & Related papers  (2025-02-18T14:32:44Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• Listening to the Wise Few: Select-and-Copy Attention Heads for Multiple-Choice QA [19.78468832417275]
  We introduce new scores that better capture and reveal model's underlying knowledge. Based on these scores, our method improves knowledge extraction, yielding up to 16% gain for LLaMA2-7B. The accuracy on a simple synthetic dataset, where the model explicitly knows the right answer, increases by almost 60%.
  arXiv  Detail & Related papers  (2024-10-03T09:53:48Z)
• Statistical Knowledge Assessment for Large Language Models [79.07989821512128]
  Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? We propose KaRR, a statistical approach to assess factual knowledge for LLMs. Our results reveal that the knowledge in LLMs with the same backbone architecture adheres to the scaling law, while tuning on instruction-following data sometimes compromises the model's capability to generate factually correct text reliably.
  arXiv  Detail & Related papers  (2023-05-17T18:54:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.