Building a Silver-Standard Dataset from NICE Guidelines for Clinical LLMs

• URL: http://arxiv.org/abs/2511.01053v1
• Date: Sun, 02 Nov 2025 19:13:37 GMT
• Title: Building a Silver-Standard Dataset from NICE Guidelines for Clinical LLMs
• Abstract summary: This study introduces a validated dataset derived from publicly available guidelines across multiple diagnoses.<n>The dataset was created with the help of GPT and contains realistic patient scenarios, as well as clinical questions.
• Score: 5.081034062303304
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used in healthcare, yet standardised benchmarks for evaluating guideline-based clinical reasoning are missing. This study introduces a validated dataset derived from publicly available guidelines across multiple diagnoses. The dataset was created with the help of GPT and contains realistic patient scenarios, as well as clinical questions. We benchmark a range of recent popular LLMs to showcase the validity of our dataset. The framework supports systematic evaluation of LLMs' clinical utility and guideline adherence.

Related papers

• Future Querying: Can LLMs Serve as Implicit Medical World Models? [3.3341114691426257]
  We introduce future querying, a paradigm that probes whether large language models (LLMs) can function as implicit medical world models.<n>Our framework operates on unstructured clinical documentation using endpoint-agnostic training.<n>We show that small, locally fine-tuned open-weight models can match or approach larger proprietary systems.
  arXiv  Detail & Related papers  (2026-08-24T13:41:12Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• Harnessing Large Language Models for Precision Querying and Retrieval-Augmented Knowledge Extraction in Clinical Data Science [3.4325249294405555]
  This study applies Large Language Models (LLMs) to two foundational Electronic Health Record (EHR) data science tasks.<n>We test the ability of LLMs to interact accurately with large structured datasets for analytics.<n>We present a flexible evaluation framework that automatically generates synthetic question and answer pairs tailored to the characteristics of each dataset or task.
  arXiv  Detail & Related papers  (2026-01-28T14:57:36Z)
• Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines [31.28422553778395]
  We introduce EPAG, a benchmark dataset and framework designed for evaluating the pre-consultation ability of LLMs.<n>Small open-source models fine-tuned with a well-curated, task-specific dataset can outperform frontier LLMs in pre-consultation.<n>We find that increased amount of HPI does not necessarily lead to improved diagnostic performance.
  arXiv  Detail & Related papers  (2026-01-07T06:15:21Z)
• ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning [58.01333341218153]
  We propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues.<n>Our method generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent.<n>Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-29T12:58:58Z)
• Timely Clinical Diagnosis through Active Test Selection [49.091903570068155]
  We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) to better emulate real-world diagnostic reasoning.<n>LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data.<n>We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use.
  arXiv  Detail & Related papers  (2025-10-21T18:10:45Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• Cohort Discovery: A Survey on LLM-Assisted Clinical Trial Recruitment [21.800829039344876]
  We are the first to analyze the task of trial-patient matching and contextualize emerging LLM-based approaches in clinical trial recruitment.<n>We critically examine existing benchmarks, approaches and evaluation frameworks.
  arXiv  Detail & Related papers  (2025-06-18T09:32:16Z)
• MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models [10.46932473088646]
  We introduce MedGUIDE, a new benchmark for evaluating Large Language Models (LLMs) on their ability to make guideline-consistent clinical decisions.<n> MedGUIDE is constructed from 55 curated NCCN decision trees across 17 cancer types.<n>We apply a two-stage quality selection process, combining expert-labeled reward models and LLM-as-a-judge ensembles across ten clinical and linguistic criteria, to select 7,747 high-quality samples.
  arXiv  Detail & Related papers  (2025-05-16T18:21:52Z)
• Large Language Models in the Clinic: A Comprehensive Benchmark [63.21278434331952]
  We build a benchmark ClinicBench to better understand large language models (LLMs) in the clinic. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. We then construct six novel datasets and clinical tasks that are complex but common in real-world practice. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2024-04-25T15:51:06Z)
• Does Biomedical Training Lead to Better Medical Performance? [7.341724934479004]
  Large Language Models (LLMs) are expected to significantly contribute to patient care, diagnostics, and administrative processes.<n>This study investigates the effect of biomedical training in the context of six practical medical tasks evaluating $25$ models.
  arXiv  Detail & Related papers  (2024-04-05T12:51:37Z)
• Developing Healthcare Language Model Embedding Spaces [0.20971479389679337]
  Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text. Three methods are assessed: traditional masked language modeling, Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR) and a novel pre-training objective utilizing metadata categories from the healthcare settings. Contrastively trained models outperform other approaches on the classification tasks, delivering strong performance from limited labeled data and with fewer model parameter updates required.
  arXiv  Detail & Related papers  (2024-03-28T19:31:32Z)
• Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching [49.78442796596806]
  We propose an innovative privacy-aware data augmentation approach for patient-trial matching (LLM-PTM) Our experiments demonstrate a 7.32% average improvement in performance using the proposed LLM-PTM method, and the generalizability to new data is improved by 12.12%.
  arXiv  Detail & Related papers  (2023-03-24T03:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.