Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction

• URL: http://arxiv.org/abs/2605.06191v1
• Date: Thu, 07 May 2026 13:05:07 GMT
• Title: Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction
• Abstract summary: This paper evaluates zero-shot and few-shot large language models (LLMs) for safety-critical clinical action extraction using the CLIP discharge-note dataset.<n>We introduce a two-stage extraction framework that decomposes discharge notes, that are written in narrative form, into fine-grained, explicitly actionable clinical tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The work in this paper evaluates zero-shot and few-shot large language models (LLMs) for safety-critical clinical action extraction using the CLIP discharge-note dataset, with particular emphasis on transitions of care and post-discharge patient safety. To manage the complexity of clinical documentation, we introduce a two-stage extraction framework that decomposes discharge notes, that are written in narrative form, into fine-grained, explicitly actionable clinical tasks through a staged prompting strategy. Our contributions include a systematic assessment of generative LLMs for clinical action extraction, a detailed comparison between general-purpose LLMs and task-specific supervised BERT-based models, and an analysis of annotation inconsistencies across different action categories. We show that contemporary LLMs achieve performance comparable to or exceeding supervised models on binary actionability detection, while supervised baselines retain a meaningful advantage on fine-grained multi-label category classification, despite the absence of task-specific fine-tuning and under strict data-privacy constraints. Qualitative error analysis reveals that many failures stem from misalignment between model reasoning and dataset annotation conventions, particularly in cases involving implicit clinical actions and rigid structural labeling rules. These results indicate that reported performance reflects model limitations due to lack of clinical reasoning, that is not captured by plain annotations. Labels without rationales make it impossible to distinguish clinical reasoning failures from annotation convention mismatches. Advancing clinical NLP requires reasoning-annotated datasets that document why specific spans are actionable, not merely which spans were labeled, enabling proper evaluation of model clinical understanding.

Related papers

• Clinical Term Extraction using Open-Source Small Language Models [2.5316486726943843]
  Open-source small language models were evaluated for ALS-relevant clinical terms in patient documentation.<n>We compared 26 open-source models using aggregate, label-level, and manual-validation multilabel classification metrics.<n>These findings support targeted hybrid extraction rather than replacement of existing rule-based methods.
  arXiv  Detail & Related papers  (2026-06-19T18:59:33Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs [63.535652574541764]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis.<n>Their application in gastrointestinal endoscopy is currently hindered by two critical limitations.<n>We propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-21T07:47:37Z)
• Measuring What VLMs Don't Say: Validation Metrics Hide Clinical Terminology Erasure in Radiology Report Generation [10.15221228043609]
  This paper investigates the use of decoding strategies that lead to high aggregate token-overlap scores despite template collapse.<n>We introduce Clinical Association Displacement (CAD), a vocabulary-level framework that quantifies shifts in demographic-based word associations in generated reports.<n>We show that deterministic decoding produces high levels of semantic erasure, while sampling generates diverse outputs but risks introducing new bias.
  arXiv  Detail & Related papers  (2026-03-02T08:59:39Z)
• AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization [6.99563009617414]
  We present AgenticSum, an inference-time framework that separates context selection, generation, verification, and targeted correction to reduce hallucinated content.<n>We evaluate AgenticSum on two public datasets, using reference-based metrics, LLM-as-a-judge assessment, and human evaluation.<n>Our results indicate that structured, agentic design with targeted correction offers an effective inference time solution to improve clinical note summarization.
  arXiv  Detail & Related papers  (2026-02-23T16:49:37Z)
• AgentScore: Autoformulation of Deployable Clinical Scoring Systems [45.88028371034407]
  We introduce AgentScore, which performs semantically guided optimization in unit-weighted clinical checklists.<n>AgentScore outperforms existing score-generation methods and achieves AUC comparable to more flexible interpretable models.<n>On two additional externally validated tasks, AgentScore achieves higher discrimination than established guideline-based scores.
  arXiv  Detail & Related papers  (2026-01-29T21:11:06Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• Measuring and Aligning Abstraction in Vision-Language Models with Medical Taxonomies [9.399546516030757]
  Vision-Language Models show strong zero-shot performance for chest X-ray classification, but standard flat metrics fail to distinguish between clinically minor and severe errors.<n>We benchmark several state-of-the-art VLMs using hierarchical metrics and introduce Catastrophic Abstraction Errors to capture cross-branch mistakes.
  arXiv  Detail & Related papers  (2026-01-21T09:58:50Z)
• Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation [61.350584471060756]
  Vision-grounded medical report generation aims to produce clinically accurate descriptions of medical images.<n>We propose Self-Supervised Anatomical Consistency Learning (SS-ACL) to align generated reports with corresponding anatomical regions.<n>SS-ACL constructs a hierarchical anatomical graph inspired by the invariant top-down inclusion structure of human anatomy.
  arXiv  Detail & Related papers  (2025-09-30T08:59:06Z)
• Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries [56.31117605097345]
  Large language models (LLMs) have shown the potential to generate accurate clinical text summaries, but still struggle with issues regarding grounding and evaluation.<n>Here, we explore a general mitigation framework using Attribute Structuring (AS), which structures the summary evaluation process.<n>AS consistently improves the correspondence between human annotations and automated metrics in clinical text summarization.
  arXiv  Detail & Related papers  (2024-03-01T21:59:03Z)
• USE-Evaluator: Performance Metrics for Medical Image Segmentation Models with Uncertain, Small or Empty Reference Annotations [5.672489398972326]
  There is a mismatch between the distributions of cases and difficulty level of segmentation tasks in public data sets compared to clinical practice. Common metrics fail to measure the impact of this mismatch, especially for clinical data sets. We study how uncertain, small, or empty reference annotations influence the value of metrics for medical image segmentation.
  arXiv  Detail & Related papers  (2022-09-26T20:40:02Z)
• Self-supervised Answer Retrieval on Clinical Notes [68.87777592015402]
  We introduce CAPR, a rule-based self-supervision objective for training Transformer language models for domain-specific passage matching. We apply our objective in four Transformer-based architectures: Contextual Document Vectors, Bi-, Poly- and Cross-encoders. We report that CAPR outperforms strong baselines in the retrieval of domain-specific passages and effectively generalizes across rule-based and human-labeled passages.
  arXiv  Detail & Related papers  (2021-08-02T10:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.