Auditing Evidence Use in Medical LLM Diagnosis

• URL: http://arxiv.org/abs/2607.20848v1
• Date: Thu, 23 Jul 2026 02:20:15 GMT
• Title: Auditing Evidence Use in Medical LLM Diagnosis
• Abstract summary: We present a behavioral audit of evidence use in medical diagnosis.<n>For each case, we decompose patient information into evidence units, score candidate diagnoses under controlled evidence subsets, and mine low-order interactions in diagnostic margins.<n>Because medical evidence is diagnosis-relative, the audit separates interaction discovery from failure assignment.
• Score: 10.934702002803434
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical LLMs are often evaluated by whether they select the correct diagnosis, but diagnostic accuracy alone does not show whether the model used the case evidence appropriately. We present a behavioral audit of evidence use in medical diagnosis. For each case, we decompose patient information into evidence units, score candidate diagnoses under controlled evidence subsets, and mine low-order interactions in diagnostic margins. Because medical evidence is diagnosis-relative, the audit separates interaction discovery from failure assignment: large or negative interactions can reflect plausible differential diagnosis, while suspicious interactions require robustness checks and clinical review. We evaluate five open-weight LLMs on DDXPlus, CupCase, and MedCase. Across datasets, faithful support and differential conflict or cancellation account for most interaction strength, showing that many evidence interactions are clinically plausible rather than failures. In a DDXPlus-focused blinded five-reviewer 130-item enriched review sample, invalid or shortcut-like cases concentrate in negated or absent findings and clinically local evidence. These results show that accuracy can hide candidate evidence-use failures and motivate role-aware audits for medical LLM evaluation.

Related papers

• EviDx: Evidence-Aware Active Diagnosis with Scaffolded LLM Agents [13.21048121687754]
  We introduce EviDx, an evidence-aware active diagnosis framework.<n>It pairs patient-specific diagnostic environments with a clinical diagnostic scaffold and an observer-guided runtime harness.<n>EviDx improves diagnostic performance and process stability while revealing model-dependent capability boundaries.
  arXiv  Detail & Related papers  (2026-08-25T13:54:30Z)
• CDEG: Learning Decision-Critical Evidence for Long-Horizon Diagnostic Agents [21.40725112789917]
  We introduce CDEG, a graph-based framework that learns reusable decision-critical evidence from historical diagnostic trajectories.<n>We show that CDEG consistently improves diagnostic performance, achieving up to an 11.5% accuracy gain over vanilla agents.
  arXiv  Detail & Related papers  (2026-08-24T07:30:22Z)
• MedReaMM: Evaluating Large Multimodal Models on Expert-Level Clinical Diagnostic Synthesis [61.266743965495884]
  We introduce MedReaMM, a benchmark designed to evaluate models' ability to synthesize heterogeneous clinical evidence.<n>MedReaMM comprises 625 expert-validated cases with an average of 2.79 medical images per case and a total of 1,042 standardized diagnoses annotated with ICD-11 codes.<n>We evaluate 23 Large Multimodal Models (LMMs) and find that most achieve diagnostic accuracy scores below 50%, underscoring a substantial gap in multimodal diagnostic synthesis capability.
  arXiv  Detail & Related papers  (2026-08-23T09:37:53Z)
• Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases [59.5328455571863]
  ClinMM-Bench is the largest multi-turn multimodal clinical diagnostic evaluation benchmark to date.<n>It contains 1,089 challenging real-world clinical cases and 3,760 medical images across eight specialties.<n> proprietary models achieved the highest overall diagnostic accuracy, but the proportion of completely correct diagnoses remained limited.
  arXiv  Detail & Related papers  (2026-07-28T16:19:03Z)
• DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs [21.52456139726765]
  Most medical AI benchmarks reveal the relevant context upfront and score only the final answer.<n>We introduce DDX-TRACE, a physician-adjudicated benchmark for multimodal neuroradiology.<n>It evaluates diagnostic trajectories under hidden evidence over 211 challenging cases.
  arXiv  Detail & Related papers  (2026-05-22T13:41:10Z)
• MultiDx: A Multi-Source Knowledge Integration Framework towards Diagnostic Reasoning [66.94527468532843]
  We propose MultiDx, a two-stage diagnostic reasoning framework.<n>It first generates suspected diagnoses and reasoning paths by leveraging knowledge from web search, SOAP-formatted case, and clinical case database.<n>Then it integrates multi-perspective evidence through matching, voting, and differential diagnosis to generate the final prediction.
  arXiv  Detail & Related papers  (2026-04-27T08:46:29Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Improving Clinical Diagnosis with Counterfactual Multi-Agent Reasoning [16.709358805587506]
  Clinical diagnosis is a complex reasoning process in which clinicians gather evidence, form hypotheses, and test them against alternative explanations.<n>We propose a counterfactual multi-agent diagnostic framework inspired by clinician training that makes hypothesis testing explicit and evidence-grounded.
  arXiv  Detail & Related papers  (2026-03-29T19:14:26Z)
• MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis [13.241795322837861]
  We introduce MedEinst, a counterfactual benchmark with 5,383 paired clinical cases across 49 diseases.<n>We measure susceptibility via Bias Trap Rate--probability of misdiagnosing traps despite correctly diagnosing controls.
  arXiv  Detail & Related papers  (2026-01-10T17:39:25Z)
• ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning [58.01333341218153]
  We propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues.<n>Our method generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent.<n>Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-29T12:58:58Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• MedCaseReasoning: Evaluating and learning diagnostic reasoning from clinical case reports [49.00805568780791]
  We introduce MedCaseReasoning, the first open-access dataset for evaluating Large Language Models (LLMs) on their ability to align with clinician-authored diagnostic reasoning.<n>The dataset includes 14,489 diagnostic question-and-answer cases, each paired with detailed reasoning statements.<n>We evaluate state-of-the-art reasoning LLMs on MedCaseReasoning and find significant shortcomings in their diagnoses and reasoning.
  arXiv  Detail & Related papers  (2025-05-16T22:34:36Z)
• Towards Reducing Diagnostic Errors with Interpretable Risk Prediction [18.474645862061426]
  We propose a method to use LLMs to identify pieces of evidence in patient EHR data that indicate increased or decreased risk of specific diagnoses. Our ultimate aim is to increase access to evidence and reduce diagnostic errors.
  arXiv  Detail & Related papers  (2024-02-15T17:05:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.