Dialogue to Question Generation for Evidence-based Medical Guideline Agent Development

• URL: http://arxiv.org/abs/2603.23937v1
• Date: Wed, 25 Mar 2026 04:58:53 GMT
• Title: Dialogue to Question Generation for Evidence-based Medical Guideline Agent Development
• Abstract summary: Evidence-based medicine (EBM) is central to high-quality care, but remains difficult to implement in fast-paced primary care settings.<n>We investigate the feasibility of using large language models (LLMs) as ambient assistants that surface targeted, evidence-based questions during physician-patient encounters.
• Score: 36.38151630332192
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Evidence-based medicine (EBM) is central to high-quality care, but remains difficult to implement in fast-paced primary care settings. Physicians face short consultations, increasing patient loads, and lengthy guideline documents that are impractical to consult in real time. To address this gap, we investigate the feasibility of using large language models (LLMs) as ambient assistants that surface targeted, evidence-based questions during physician-patient encounters. Our study focuses on question generation rather than question answering, with the aim of scaffolding physician reasoning and integrating guideline-based practice into brief consultations. We implemented two prompting strategies, a zero-shot baseline and a multi-stage reasoning variant, using Gemini 2.5 as the backbone model. We evaluated on a benchmark of 80 de-identified transcripts from real clinical encounters, with six experienced physicians contributing over 90 hours of structured review. Results indicate that while general-purpose LLMs are not yet fully reliable, they can produce clinically meaningful and guideline-relevant questions, suggesting significant potential to reduce cognitive burden and make EBM more actionable at the point of care.

Related papers

• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows [7.354281305130301]
  Large language models (LLMs) are increasingly considered for use in clinical consultation tasks.<n>We propose AIPatient Arena, an EHRs-grounded evaluation framework for assessing the clinical utility of LLMs.
  arXiv  Detail & Related papers  (2026-06-16T03:35:17Z)
• EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries [13.472329890204831]
  We introduce EHRNote-ChatQA, the first benchmark for evidence-grounded multi-turn clinical question answering over patients' multiple discharge summaries.<n>Built from de-identified MIMIC-IV discharge summaries, EHRNote-ChatQA contains 967 patient-level multi-turn samples spanning one to five notes and 16,072 medical-expert-verified QA pairs.
  arXiv  Detail & Related papers  (2026-06-14T10:35:19Z)
• When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering [10.478229476671949]
  Most medical large language models (LLMs) are trained to encode common, guideline-focused medical knowledge in their parameters.<n>We introduce OGCaReBench, a free-form retrieval-focused benchmark aimed at evaluating LLMs at answering clinical questions that require going beyond typical guidelines.<n>Our experiments reveal that even the best-performing baseline (GPT-5.2) correctly answers only 56% of our benchmark with specialized models only reaching 42%.
  arXiv  Detail & Related papers  (2026-05-20T23:04:48Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• MedArena: Comparing LLMs for Medicine-in-the-Wild Clinician Preferences [50.71326426975699]
  We present MedArena, an interactive evaluation platform for medical large language models (LLMs)<n>Given a clinician-provided query, MedArena presents responses from two randomly selected models and asks the user to select the preferred response.<n>Out of 1571 preferences collected across 12 LLMs up to November 1, 2025, Gemini 2.0 Flash Thinking, Gemini 2.5 Pro, and GPT-4o were the top three models by Bradley-Terry rating.
  arXiv  Detail & Related papers  (2026-03-13T22:30:26Z)
• The Illusion of Clinical Reasoning: A Benchmark Reveals the Pervasive Gap in Vision-Language Models for Clinical Competency [38.68458713626548]
  Current benchmarks fail to capture the integrated, multimodal reasoning essential for real-world patient care.<n>This benchmark assesses models across 7 tasks that mirror the clinical reasoning pathway.<n>Current artificial intelligence models are not yet clinically competent for complex, multimodal reasoning.
  arXiv  Detail & Related papers  (2025-12-25T03:33:22Z)
• Orchestrator Multi-Agent Clinical Decision Support System for Secondary Headache Diagnosis in Primary Care [13.520457515792534]
  We present a large language model (LLM)-based multi-agent clinical decision support system built on an orchestrator-specialist architecture.<n>The system decomposes diagnosis into seven domain-specialized agents, each producing a structured and evidence-grounded rationale.<n>We evaluated the multi-agent system using 90 expert-validated secondary headache cases and compared its performance with a single-LLM baseline.
  arXiv  Detail & Related papers  (2025-12-03T19:26:12Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering [1.0262304700896199]
  We present CounselBench, a large-scale benchmark developed with 100 mental health professionals to evaluate and stress-test large language models (LLMs)<n>The first component, CounselBench-EVAL, contains 2,000 expert evaluations of answers from GPT-4, LLaMA 3, Gemini, and human therapists on patient questions from the public forum CounselChat.<n>Expert evaluations show that while LLMs achieve high scores on several dimensions, they also exhibit recurring issues, including unconstructive feedback, overgeneralization, and limited personalization or relevance.
  arXiv  Detail & Related papers  (2025-06-10T08:53:06Z)
• Towards Conversational AI for Disease Management [29.189384095061722]
  Articulate Medical Intelligence Explorer (AMIE) is an agentic system optimised for clinical management and dialogue.<n>AMIE is non-inferior to PCPs in management reasoning as assessed by specialist physicians.<n>AMIE's strong performance across evaluations marks a significant step towards conversational AI as a tool in disease management.
  arXiv  Detail & Related papers  (2025-03-08T05:48:58Z)
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts [50.02627258858336]
  Large language models (LLMs) often struggle with open-ended medical questions.<n>We propose a novel approach utilizing structured medical reasoning.<n>Our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models.
  arXiv  Detail & Related papers  (2025-03-05T05:24:55Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.