Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines

• URL: http://arxiv.org/abs/2601.03627v2
• Date: Thu, 08 Jan 2026 01:47:05 GMT
• Title: Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines
• Abstract summary: We introduce EPAG, a benchmark dataset and framework designed for evaluating the pre-consultation ability of LLMs.<n>Small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation.<n>We find that increased amount of HPI does not necessarily lead to improved diagnostic performance.
• Score: 31.28422553778395
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce EPAG, a benchmark dataset and framework designed for Evaluating the Pre-consultation Ability of LLMs using diagnostic Guidelines. LLMs are evaluated directly through HPI-diagnostic guideline comparison and indirectly through disease diagnosis. In our experiments, we observe that small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation. Additionally, we find that increased amount of HPI (History of Present Illness) does not necessarily lead to improved diagnostic performance. Further experiments reveal that the language of pre-consultation influences the characteristics of the dialogue. By open-sourcing our dataset and evaluation pipeline on https://github.com/seemdog/EPAG, we aim to contribute to the evaluation and further development of LLM applications in real-world clinical settings.

Related papers

• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• EviCare: Enhancing Diagnosis Prediction with Deep Model-Guided Evidence for In-Context Reasoning [62.61394722212386]
  EviCare is an in-context reasoning framework that integrates deep model guidance into large language models.<n>It consistently outperforms both LLM-only and deep model-only baselines in experiments on two real-world EHR benchmarks.
  arXiv  Detail & Related papers  (2026-04-12T04:35:14Z)
• Building a Silver-Standard Dataset from NICE Guidelines for Clinical LLMs [5.081034062303304]
  This study introduces a validated dataset derived from publicly available guidelines across multiple diagnoses.<n>The dataset was created with the help of GPT and contains realistic patient scenarios, as well as clinical questions.
  arXiv  Detail & Related papers  (2025-11-02T19:13:37Z)
• Timely Clinical Diagnosis through Active Test Selection [49.091903570068155]
  We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) to better emulate real-world diagnostic reasoning.<n>LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data.<n>We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use.
  arXiv  Detail & Related papers  (2025-10-21T18:10:45Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Cohort Discovery: A Survey on LLM-Assisted Clinical Trial Recruitment [21.800829039344876]
  We are the first to analyze the task of trial-patient matching and contextualize emerging LLM-based approaches in clinical trial recruitment.<n>We critically examine existing benchmarks, approaches and evaluation frameworks.
  arXiv  Detail & Related papers  (2025-06-18T09:32:16Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• Improving Interactive Diagnostic Ability of a Large Language Model Agent Through Clinical Experience Learning [17.647875658030006]
  This study investigates the underlying mechanisms behind the performance degradation phenomenon.<n>We developed a plug-and-play method enhanced (PPME) LLM agent, leveraging over 3.5 million electronic medical records from Chinese and American healthcare facilities.<n>Our approach integrates specialized models for initial disease diagnosis and inquiry into the history of the present illness, trained through supervised and reinforcement learning techniques.
  arXiv  Detail & Related papers  (2025-02-24T06:24:20Z)
• Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filtering [66.5524727179286]
  NOVA is a framework designed to identify high-quality data that aligns well with the learned knowledge to reduce hallucinations.<n>It includes Internal Consistency Probing (ICP) and Semantic Equivalence Identification (SEI) to measure how familiar the LLM is with instruction data.<n>To ensure the quality of selected samples, we introduce an expert-aligned reward model, considering characteristics beyond just familiarity.
  arXiv  Detail & Related papers  (2025-02-11T08:05:56Z)
• Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval [1.986227187900497]
  We conduct experiments using the MIMIC dataset to explore the impact of the prompt structure, instruction, context, and demonstration.<n>Our findings show that optimal feature selection and serialization methods can enhance task performance by up to 26.79%.<n>In-context learning setups with relevant example selection improve data extraction performance by 5.95%.
  arXiv  Detail & Related papers  (2025-01-16T08:52:50Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges [18.56314471146199]
  Large volume of notes often associated with patients together with time constraints renders manually identifying relevant evidence practically infeasible. We propose and evaluate a zero-shot strategy for using LLMs as a mechanism to efficiently retrieve and summarize unstructured evidence in patient EHR.
  arXiv  Detail & Related papers  (2023-09-08T18:44:47Z)
• Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching [49.78442796596806]
  We propose an innovative privacy-aware data augmentation approach for patient-trial matching (LLM-PTM) Our experiments demonstrate a 7.32% average improvement in performance using the proposed LLM-PTM method, and the generalizability to new data is improved by 12.12%.
  arXiv  Detail & Related papers  (2023-03-24T03:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.