Exploring Large Language Models in Healthcare: Insights into Corpora Sources, Customization Strategies, and Evaluation Metrics

• URL: http://arxiv.org/abs/2502.11861v1
• Date: Mon, 17 Feb 2025 14:53:23 GMT
• Title: Exploring Large Language Models in Healthcare: Insights into Corpora Sources, Customization Strategies, and Evaluation Metrics
• Authors: Shuqi Yang, Mingrui Jing, Shuai Wang, Jiaxin Kou, Manfei Shi, Weijie Xing, Yan Hu, Zheng Zhu,
• Abstract summary: This study reviewed the use of Large Language Models (LLMs) in healthcare, focusing on their training corpora, customization techniques, and evaluation metrics.<n>Four types of corpora were used: clinical resources, literature, open-source datasets, and web-crawled data.<n>The reliance on unverified or unstructured data highlighted the need for better integration of evidence-based clinical guidelines.
• Score: 21.114147435973468
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This study reviewed the use of Large Language Models (LLMs) in healthcare, focusing on their training corpora, customization techniques, and evaluation metrics. A systematic search of studies from 2021 to 2024 identified 61 articles. Four types of corpora were used: clinical resources, literature, open-source datasets, and web-crawled data. Common construction techniques included pre-training, prompt engineering, and retrieval-augmented generation, with 44 studies combining multiple methods. Evaluation metrics were categorized into process, usability, and outcome metrics, with outcome metrics divided into model-based and expert-assessed outcomes. The study identified critical gaps in corpus fairness, which contributed to biases from geographic, cultural, and socio-economic factors. The reliance on unverified or unstructured data highlighted the need for better integration of evidence-based clinical guidelines. Future research should focus on developing a tiered corpus architecture with vetted sources and dynamic weighting, while ensuring model transparency. Additionally, the lack of standardized evaluation frameworks for domain-specific models called for comprehensive validation of LLMs in real-world healthcare settings.

Related papers

• Large Language Models for Healthcare Text Classification: A Systematic Review [4.8342038441006805]
  Large Language Models (LLMs) have fundamentally transformed approaches to Natural Language Processing (NLP) In healthcare, accurate and cost-efficient text classification is crucial, whether for clinical notes analysis, diagnosis coding, or any other task. Numerous studies have been conducted to leverage LLMs for automated healthcare text classification.
  arXiv  Detail & Related papers  (2025-03-03T04:16:13Z)
• ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports [3.0363830583066713]
  This paper presents the results of our project, which aims to leverage Large Language Models (LLMs) to extract structured information from unstructured clinical reports.<n>We developed a workflow with a user interface and evaluated LLMs of varying sizes through prompting strategies and fine-tuning.<n>Our results show that fine-tuned smaller models match or surpass larger counterparts in performance, offering efficiency for resource-limited settings.
  arXiv  Detail & Related papers  (2025-02-08T16:44:56Z)
• Representation Learning of Structured Data for Medical Foundation Models [29.10129199884847]
  We introduce the UniStruct architecture to design a multimodal medical foundation model of unstructured text and structured data. Our approach is validated through model pre-training on both an extensive internal medical database and a public repository of structured medical records.
  arXiv  Detail & Related papers  (2024-10-17T09:02:28Z)
• Named Clinical Entity Recognition Benchmark [2.9332007863461893]
  This report introduces a Named Clinical Entity Recognition Benchmark. It addresses the crucial natural language processing (NLP) task of extracting structured information from clinical narratives. The leaderboard provides a standardized platform for assessing diverse language models.
  arXiv  Detail & Related papers  (2024-10-07T14:00:18Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry [2.1717945745027425]
  Large Language Models (LLMs) have evolved significantly, impacting various industries with their advanced capabilities in language understanding and generation. This comprehensive survey delineates the extensive application and requisite evaluation of LLMs within healthcare. Our survey is structured to provide an in-depth analysis of LLM applications across clinical settings, medical text data processing, research, education, and public health awareness.
  arXiv  Detail & Related papers  (2024-04-24T09:55:24Z)
• Evaluating the Fairness of the MIMIC-IV Dataset and a Baseline Algorithm: Application to the ICU Length of Stay Prediction [65.268245109828]
  This paper uses the MIMIC-IV dataset to examine the fairness and bias in an XGBoost binary classification model predicting the ICU length of stay. The research reveals class imbalances in the dataset across demographic attributes and employs data preprocessing and feature extraction. The paper concludes with recommendations for fairness-aware machine learning techniques for mitigating biases and the need for collaborative efforts among healthcare professionals and data scientists.
  arXiv  Detail & Related papers  (2023-12-31T16:01:48Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Development and validation of a natural language processing algorithm to pseudonymize documents in the context of a clinical data warehouse [53.797797404164946]
  The study highlights the difficulties faced in sharing tools and resources in this domain. We annotated a corpus of clinical documents according to 12 types of identifying entities. We build a hybrid system, merging the results of a deep learning model as well as manual rules.
  arXiv  Detail & Related papers  (2023-03-23T17:17:46Z)
• A Survey on Text Classification: From Shallow to Deep Learning [83.47804123133719]
  The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification.
  arXiv  Detail & Related papers  (2020-08-02T00:09:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.