Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization

• URL: http://arxiv.org/abs/2309.07430v5
• Date: Thu, 11 Apr 2024 19:47:18 GMT
• Title: Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization
• Authors: Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerova, Nidhi Rohatgi, Poonam Hosamani, William Collins, Neera Ahuja, Curtis P. Langlotz, Jason Hom, Sergios Gatidis, John Pauly, Akshay S. Chaudhari,
• Abstract summary: Large language models (LLMs) have shown promise in natural language processing (NLP), but their effectiveness on a diverse range of clinical summarization tasks remains unproven. In this study, we apply adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks. A clinical reader study with ten physicians evaluates summary, completeness, correctness, and conciseness; in a majority of cases, summaries from our best adapted LLMs are either equivalent (45%) or superior (36%) compared to summaries from medical experts.
• Score: 8.456700096020601
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP), their effectiveness on a diverse range of clinical summarization tasks remains unproven. In this study, we apply adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Quantitative assessments with syntactic, semantic, and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with ten physicians evaluates summary completeness, correctness, and conciseness; in a majority of cases, summaries from our best adapted LLMs are either equivalent (45%) or superior (36%) compared to summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.

Related papers

• Demystifying Large Language Models for Medicine: A Primer [50.83806796466396]
  Large language models (LLMs) represent a transformative class of AI tools capable of revolutionizing various aspects of healthcare. This tutorial aims to equip healthcare professionals with the tools necessary to effectively integrate LLMs into clinical practice.
  arXiv  Detail & Related papers  (2024-10-24T15:41:56Z)
• CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios [50.032101237019205]
  CliMedBench is a comprehensive benchmark with 14 expert-guided core clinical scenarios. The reliability of this benchmark has been confirmed in several ways.
  arXiv  Detail & Related papers  (2024-10-04T15:15:36Z)
• DiReCT: Diagnostic Reasoning for Clinical Notes via Large Language Models [32.85606857702375]
  We aim at evaluating the reasoning ability and interpretability of large language models (LLMs) compared to human doctors. The diagnostic reasoning dataset for clinical notes (DiReCT) contains 511 clinical notes, each meticulously annotated by physicians.
  arXiv  Detail & Related papers  (2024-08-04T05:15:02Z)
• SemioLLM: Assessing Large Language Models for Semiological Analysis in Epilepsy Research [45.2233252981348]
  Large Language Models have shown promising results in their ability to encode general medical knowledge. We test the ability of state-of-the-art LLMs to leverage their internal knowledge and reasoning for epilepsy diagnosis.
  arXiv  Detail & Related papers  (2024-07-03T11:02:12Z)
• Large Language Models in the Clinic: A Comprehensive Benchmark [63.21278434331952]
  We build a benchmark ClinicBench to better understand large language models (LLMs) in the clinic. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. We then construct six novel datasets and clinical tasks that are complex but common in real-world practice. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2024-04-25T15:51:06Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review [16.008511195589925]
  Large language models (LLMs) have shown promising capabilities in mimicking human-level language comprehension and reasoning. This paper provides a comprehensive review on the applications and implications of LLMs in medicine.
  arXiv  Detail & Related papers  (2023-11-03T13:51:36Z)
• Deciphering Diagnoses: How Large Language Models Explanations Influence Clinical Decision Making [0.0]
  Large Language Models (LLMs) are emerging as a promising tool to generate plain-text explanations for medical decisions. This study explores the effectiveness and reliability of LLMs in generating explanations for diagnoses based on patient complaints.
  arXiv  Detail & Related papers  (2023-10-03T00:08:23Z)
• Self-Verification Improves Few-Shot Clinical Information Extraction [73.6905567014859]
  Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning. They still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs.
  arXiv  Detail & Related papers  (2023-05-30T22:05:11Z)
• Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding [12.128991867050487]
  Large language models (LLMs) have made significant progress in various domains, including healthcare. In this study, we evaluate state-of-the-art LLMs within the realm of clinical language understanding tasks.
  arXiv  Detail & Related papers  (2023-04-09T16:31:47Z)
• SPeC: A Soft Prompt-Based Calibration on Performance Variability of Large Language Model in Clinical Notes Summarization [50.01382938451978]
  We introduce a model-agnostic pipeline that employs soft prompts to diminish variance while preserving the advantages of prompt-based summarization. Experimental findings indicate that our method not only bolsters performance but also effectively curbs variance for various language models.
  arXiv  Detail & Related papers  (2023-03-23T04:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.