MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

• URL: http://arxiv.org/abs/2605.30295v1
• Date: Thu, 28 May 2026 17:42:43 GMT
• Title: MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings
• Abstract summary: Large language models (LLMs) show promise for clinical reasoning and decision support.<n>We introduce a pipeline for generating clinically realistic HL7 FHIR R4 bundles from unstructured text.<n>Applying this approach to MedCaseReasoning, we construct MedCase-Structured, a synthetic dataset aligned with clinician-authored diagnostic cases.
• Score: 2.759077490183459
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent settings remains limited. Existing benchmarks often rely on static datasets or unstructured inputs that do not reflect the structured, interoperable data formats used in clinical systems. We introduce a pipeline for generating clinically realistic HL7 FHIR R4 bundles from unstructured text, enabling controllable evaluation of clinical decision support systems. The pipeline combines staged LLM generation with terminology-grounded validation and repair to reduce hallucinated codes and enforce structural and semantic consistency. Applying this approach to MedCaseReasoning, we construct MedCase-Structured, a synthetic dataset aligned with clinician-authored diagnostic cases, achieving valid FHIR generation for 82.5% of cases. Evaluation on MedCase-Structured reveals consistently lower diagnostic accuracy for LLMs on structured FHIR inputs than with plain text, highlighting the importance of deployment-aligned benchmarking.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• LLM-Orchestrated Conformance Checking in Stroke Care Without Computer-Interpretable Guidelines [35.18016233072556]
  Conformance checking in healthcare seeks to assess whether patient care pathways adhere to clinical guidelines.<n>This work introduces a modular framework to support medical conformance checking directly from unstructured clinical and guideline texts.
  arXiv  Detail & Related papers  (2026-06-08T13:44:03Z)
• KEPIL: Knowledge-Enhanced Prompt-Image Learning for Prompt-Robust Disease Detection [6.447908430647854]
  radiological findings are long-tailed in practice, leaving some conditions underrepresented and making zero-shot inference essential.<n>We present textitKEPIL, a prompt-robust framework that integrates curated medical knowledge to stabilize zero-shot generalization.
  arXiv  Detail & Related papers  (2026-05-09T19:29:01Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning [58.01333341218153]
  We propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues.<n>Our method generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent.<n>Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-29T12:58:58Z)
• CNSight: Evaluation of Clinical Note Segmentation Tools [3.673249612734457]
  We evaluate rule-based baselines, domain-specific transformer models, and large language models for clinical note segmentation using a curated dataset of 1,000 notes from MIMIC-IV.<n>Our experiments show that large API-based models achieve the best overall performance, with GPT-5-mini reaching a best average F1 of 72.4 across sentence-level and freetext segmentation.
  arXiv  Detail & Related papers  (2025-12-28T05:40:15Z)
• Timely Clinical Diagnosis through Active Test Selection [49.091903570068155]
  We propose ACTMED (Adaptive Clinical Test selection via Model-based Experimental Design) to better emulate real-world diagnostic reasoning.<n>LLMs act as flexible simulators, generating plausible patient state distributions and supporting belief updates without requiring structured, task-specific training data.<n>We evaluate ACTMED on real-world datasets and show it can optimize test selection to improve diagnostic accuracy, interpretability, and resource use.
  arXiv  Detail & Related papers  (2025-10-21T18:10:45Z)
• Infherno: End-to-end Agent-based FHIR Resource Synthesis from Free-form Clinical Notes [37.88754205269813]
  We propose an end-to-end framework powered by LLM agents, code execution, and healthcare terminology database tools to address these issues.<n>Our solution, called Infherno, is designed to adhere to the FHIR document schema and competes well with a human baseline in predicting FHIR resources from unstructured text.
  arXiv  Detail & Related papers  (2025-07-16T14:06:51Z)
• CLI-RAG: A Retrieval-Augmented Framework for Clinically Structured and Context Aware Text Generation with LLMs [0.1578515540930834]
  We introduce CLI-RAG (Clinically Informed Retrieval-Augmented Generation), a domain-specific framework for structured and clinically grounded text generation.<n>It incorporates a novel hierarchical chunking strategy that respects clinical document structure and introduces a task-specific dual-stage retrieval mechanism.<n>We apply the system to generate structured progress notes for individual hospital visits using 15 clinical note types from the MIMIC-III dataset.
  arXiv  Detail & Related papers  (2025-07-09T10:13:38Z)
• SemioLLM: Evaluating Large Language Models for Diagnostic Reasoning from Unstructured Clinical Narratives in Epilepsy [45.2233252981348]
  Large Language Models (LLMs) have been shown to encode clinical knowledge.<n>We present SemioLLM, an evaluation framework that benchmarks 6 state-of-the-art models.<n>We show that most LLMs are able to accurately and confidently generate probabilistic predictions of seizure onset zones in the brain.
  arXiv  Detail & Related papers  (2024-07-03T11:02:12Z)
• TREEMENT: Interpretable Patient-Trial Matching via Personalized Dynamic Tree-Based Memory Network [54.332862955411656]
  Clinical trials are critical for drug development but often suffer from expensive and inefficient patient recruitment. In recent years, machine learning models have been proposed for speeding up patient recruitment via automatically matching patients with clinical trials. We introduce a dynamic tree-based memory network model named TREEMENT to provide accurate and interpretable patient trial matching.
  arXiv  Detail & Related papers  (2023-07-19T12:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.