MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations

• URL: http://arxiv.org/abs/2609.12851v1
• Date: Fri, 11 Sep 2026 13:41:53 GMT
• Title: MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
• Abstract summary: Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations.<n>We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties.
• Score: 37.085079820080644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical benchmarks are dominated by single-turn, multiple-choice clinical cases that poorly reflect real consultations. Practically, clinicians elicit evidence interactively and patient communication varies widely. We introduce MedRoundsQA, a multi-turn diagnostic benchmark derived from 1,387 board-exam cases across 17 specialties. Each case is converted into a structured 24-slot clinical record, and then instantiated as controlled doctor-patient dual-agent dialogues under varying patient personas, with the underlying clinical content held fixed. We further classify cases by difficulty using model-based uncertainty to enable easy-to-hard analysis. Evaluations of fifteen LLM doctor agents show that (i) moving from a single-turn diagnosis on the standardized records to multi-turn consultations causes large degradations of roughly 13-39 points; (ii) more turns reliably improves question relevance, but diagnostic accuracy exhibits diminishing returns and typically plateaus after 6-12 turns; and (iii) patient persona differences can shift diagnosis accuracy by about 7-8 points (lowest to highest education), highlighting equity risks that single-turn benchmarks miss.

Related papers

• MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis [61.266743965495884]
  We introduce MedReaMM, a benchmark designed to evaluate models' ability to synthesize heterogeneous clinical evidence.<n>MedReaMM comprises 625 expert-validated cases with an average of 2.79 medical images per case and a total of 1,042 standardized diagnoses annotated with ICD-11 codes.<n>We evaluate 23 Large Multimodal Models (LMMs) and find that most achieve diagnostic accuracy scores below 50%, underscoring a substantial gap in multimodal diagnostic synthesis capability.
  arXiv  Detail & Related papers  (2026-08-23T09:37:53Z)
• Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases [59.5328455571863]
  ClinMM-Bench is the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date.<n>It contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties.<n> proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited.
  arXiv  Detail & Related papers  (2026-07-28T16:19:03Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• One Panel Does Not Fit All: Case-Adaptive Multi-Agent Deliberation for Clinical Prediction [10.760929240872564]
  We propose CAMP (Case-Adaptive Multi-agent Panel), where an attending-physician agent dynamically assembles a specialist panel tailored to each case's diagnostic uncertainty.<n>A hybrid router directs each diagnosis through strong consensus, fallback to the attending physician's judgment, or evidence-based arbitration that weighs argument quality over vote counts.<n> CAMP consistently outperforms strong baselines while consuming fewer tokens than most competing multi-agent methods, with voting records and arbitration traces offering transparent decision audits.
  arXiv  Detail & Related papers  (2026-03-31T18:00:34Z)
• Stop Listening to Me! How Multi-turn Conversations Can Degrade Diagnostic Reasoning [12.166175637413637]
  We evaluate 17 large language models (LLMs) across three clinical datasets.<n>We develop a "stick-or-switch" evaluation framework to measure model conviction.<n>Our experiments reveal the conversation tax, where multi-turn interactions consistently degrade performance.
  arXiv  Detail & Related papers  (2026-03-12T00:14:35Z)
• LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis [14.82377002030236]
  Mental disorders are highly prevalent worldwide.<n>The shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment.<n>We present LingxiDiagBench, a large-scale multi-agent benchmark.
  arXiv  Detail & Related papers  (2026-02-10T03:46:05Z)
• Evolving Diagnostic Agents in a Virtual Clinical Environment [75.59389103511559]
  We present a framework for training large language models (LLMs) as diagnostic agents with reinforcement learning.<n>Our method acquires diagnostic strategies through interactive exploration and outcome-based feedback.<n>DiagAgent significantly outperforms 10 state-of-the-art LLMs, including DeepSeek-v3 and GPT-4o.
  arXiv  Detail & Related papers  (2025-10-28T17:19:47Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• DiagnosisArena: Benchmarking Diagnostic Reasoning for Large Language Models [25.13622249539088]
  DiagnosisArena is a benchmark designed to rigorously assess professional-level diagnostic competence.<n> DiagnosisArena consists of 1,113 pairs of segmented patient cases and corresponding diagnoses, spanning 28 medical specialties.<n>Our study reveals that even the most advanced reasoning models, o3, o1, and DeepSeek-R1, achieve only 51.12%, 31.09%, and 17.79% accuracy, respectively.
  arXiv  Detail & Related papers  (2025-05-20T09:14:53Z)
• Language models are susceptible to incorrect patient self-diagnosis in medical applications [0.0]
  We present a variety of LLMs with multiple-choice questions from U.S. medical board exams modified to include self-diagnostic reports from patients. Our findings highlight that when a patient proposes incorrect bias-validating information, the diagnostic accuracy of LLMs drop dramatically.
  arXiv  Detail & Related papers  (2023-09-17T19:56:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.