Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

• URL: http://arxiv.org/abs/2609.16023v1
• Date: Thu, 10 Sep 2026 01:18:39 GMT
• Title: Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks
• Abstract summary: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes.<n> Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative.<n>We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected.
• Score: 0.028647133890966986
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to two clinical benchmarks (HealthBench Professional and LiveMedBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form "at least one of / all of the following" as equally weighted children and regrading identical responses shifts scores by up to 15.9 percentage points on affected conversations, with disjunctive bundles inflating scores and conjunctive bundles deflating them. We also find that RIFT generally under-detects bundling on clinical rubrics, flagging 3.3% of LiveMedBench criteria as non-atomic where surface-form analysis finds structure in 25.8%.

Related papers

• MedSNIP: Building and Benchmarking Snippet-Level Granularity for Medical Fact Verification [50.13873960887466]
  We introduce MedSNIP-Bench, a benchmark for snippet-level medical fact verification.<n>We also introduce MedSNIP, an automatic snippet-generation pipeline.<n>Across MedSNIP-Bench, HealthFC, and MedHallu, snippet-level verification preserves or improves false-class F1.
  arXiv  Detail & Related papers  (2026-09-11T14:08:44Z)
• ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answering [33.27644708906731]
  Reinforcement learning with verifiable rewards has been especially effective in mathematics and coding.<n>We introduce ConRub-Med to preserve useful distinctions as rubric feedback moves from construction to policy optimization.<n>ConRub-Med ranks first on six of nine benchmarks and the highest medical and generalization averages.
  arXiv  Detail & Related papers  (2026-08-11T14:48:15Z)
• MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation [0.49454229616760287]
  Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation.<n>We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI.<n>We propose Evidence-Anchored RAG, a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective.
  arXiv  Detail & Related papers  (2026-07-15T21:56:19Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV [0.0]
  Reasoning benchmarks measure clinical performance on clean inputs.<n>We evaluate the step before reasoning: retrieval over real EHR notes, where negation, temporality, and family-versus-patient attribution can flip a correct answer to a wrong one.<n>EpiKG carries assertion label and a temporality tag with every fact in a patient knowledge graph, then routes retrieval by question intent.
  arXiv  Detail & Related papers  (2026-05-11T18:47:52Z)
• Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness [49.2667937337333]
  We stress-test this assumption for detecting incomplete patient-facing medical responses.<n>We evaluate three granularities (General-Likert, Analytical-Rubric, Dynamic-Checklist) and three backbone models across two clinician-annotated datasets.
  arXiv  Detail & Related papers  (2026-03-26T19:01:55Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems [19.880569341968023]
  Large Language Models (LLMs) are increasingly used for clinical decision support, where hallucinations and unsafe suggestions may pose direct risks to patient safety.<n>We propose a retrieval-augmented multi-agent framework designed to automate the generation of instance-specific evaluation rubrics.
  arXiv  Detail & Related papers  (2026-01-21T16:40:41Z)
• From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations [45.414878840652115]
  Large language models (LLMs) have demonstrated promising performance on medical benchmarks.<n>However, their ability to perform medical calculations remains underexplored and poorly evaluated.<n>In this work, we revisit medical calculation evaluation with a stronger focus on clinical trustworthiness.
  arXiv  Detail & Related papers  (2025-09-20T09:10:26Z)
• Dynamic Bank Learning for Semi-supervised Federated Image Diagnosis with Class Imbalance [65.61909544178603]
  We study a practical yet challenging problem of class imbalanced semi-supervised FL (imFed-Semi) This imFed-Semi problem is addressed by a novel dynamic bank learning scheme, which improves client training by exploiting class proportion information. We evaluate our approach on two public real-world medical datasets, including the intracranial hemorrhage diagnosis with 25,000 CT slices and skin lesion diagnosis with 10,015 dermoscopy images.
  arXiv  Detail & Related papers  (2022-06-27T06:51:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.