Benchmarking Clinical Decision Pathway Adherence in Large Language Models

• URL: http://arxiv.org/abs/2608.26592v1
• Date: Thu, 27 Aug 2026 04:17:12 GMT
• Title: Benchmarking Clinical Decision Pathway Adherence in Large Language Models
• Abstract summary: MEGA-CDP is a benchmark for evaluating whether medical LLMs can generate guideline-adherent CDPs.<n>It is constructed from 2,274 English and Chinese clinical practice guidelines through a guideline-to-case pipeline.<n>It supports both single-turn vignette and multi-turn interactive settings, and introduces a CDP-oriented evaluation framework.
• Score: 111.31765633748081
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Following clinical decision pathways (CDPs) defined by clinical practice guidelines is essential for safe and reliable medical decision-making. However, existing medical large language model (LLM) benchmarks mainly evaluate final-answer accuracy, providing limited evaluation of models' ability to adhere to guidelines. To address this gap, we introduce MEGA-CDP, a benchmark for evaluating whether medical LLMs can generate guideline-adherent CDPs using provided guidelines as references. MEGA-CDP is constructed from 2,274 English and Chinese clinical practice guidelines through a guideline-to-case pipeline, yielding 42,353 clinical cases with explicit reference CDPs. It supports both single-turn vignette and multi-turn interactive settings, and introduces a CDP-oriented evaluation framework for measuring pathway consistency. Experiments on 16 representative LLMs show that reliable clinical decision support remains challenging for current models, demonstrating the need for CDP-oriented evaluation and the value of MEGA-CDP for advancing guideline adherence in medical LLMs.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines [35.18016233072556]
  Conformance checking in healthcare seeks to assess whether patient care pathways adhere to clinical guidelines.<n>This work introduces a modular framework to support medical conformance checking directly from unstructured clinical and guideline texts.
  arXiv  Detail & Related papers  (2026-06-08T13:44:03Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning [24.304759042225697]
  Clinical practice guidelines (CPGs) encode evidence-based decision logic that clinicians apply by evaluating patient variables, conditional criteria, and recommendation rules.<n>We introduce a guideline-derived training pipeline that transforms CPG recommendations into executable clinical decision logic.<n>Across four clinical reasoning benchmarks, MedGuideX achieves a 10.28% relative improvement in average accuracy.
  arXiv  Detail & Related papers  (2026-05-26T05:36:05Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences [50.71326426975699]
  We present MedArena, an interactive evaluation platform for medical large language models (LLMs)<n>Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response.<n>Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating.
  arXiv  Detail & Related papers  (2026-03-13T22:30:26Z)
• From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports [13.226827332616134]
  Large language models (LLMs) have demonstrated promising performance in generating diagnostic conclusions from imaging findings.<n>This study aims to enhance the trustworthiness of LLM-generated liver MRI reports by introducing a Multi-Dimensional Credibility Assessment (MDCA) framework.
  arXiv  Detail & Related papers  (2025-10-27T04:57:20Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Refine Medical Diagnosis Using Generation Augmented Retrieval and Clinical Practice Guidelines [16.56254046507092]
  We introduce GARMLE-G, a Generation-Augmented Retrieval framework that grounds medical language model outputs in authoritative guidelines.<n>Unlike conventional Retrieval-Augmented Generation based approaches, GARMLE-G enables hallucination-free outputs by directly retrieving authoritative guideline content.<n>A prototype system for hypertension diagnosis was developed and evaluated on multiple metrics, demonstrating superior retrieval precision, semantic relevance, and clinical guideline adherence.
  arXiv  Detail & Related papers  (2025-06-22T11:31:13Z)
• MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models [10.46932473088646]
  We introduce MedGUIDE, a new benchmark for evaluating Large Language Models (LLMs) on their ability to make guideline-consistent clinical decisions.<n> MedGUIDE is constructed from 55 curated NCCN decision trees across 17 cancer types.<n>We apply a two-stage quality selection process, combining expert-labeled reward models and LLM-as-a-judge ensembles across ten clinical and linguistic criteria, to select 7,747 high-quality samples.
  arXiv  Detail & Related papers  (2025-05-16T18:21:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.