Improving Generalizability of Extracting Social Determinants of Health Using Large Language Models through Prompt-tuning

• URL: http://arxiv.org/abs/2403.12374v1
• Date: Tue, 19 Mar 2024 02:34:33 GMT
• Title: Improving Generalizability of Extracting Social Determinants of Health Using Large Language Models through Prompt-tuning
• Authors: Cheng Peng, Zehao Yu, Kaleb E Smith, Wei-Hsuan Lo-Ciganic, Jiang Bian, Yonghui Wu,
• Abstract summary: Methods based on the fine-tuning strategy have limited transfer learning ability for cross-domain applications. This study proposed a novel approach that employs a soft prompt-based learning architecture, which introduces trainable prompts to guide LLMs toward desired outputs.
• Score: 24.050346319335098
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The progress in natural language processing (NLP) using large language models (LLMs) has greatly improved patient information extraction from clinical narratives. However, most methods based on the fine-tuning strategy have limited transfer learning ability for cross-domain applications. This study proposed a novel approach that employs a soft prompt-based learning architecture, which introduces trainable prompts to guide LLMs toward desired outputs. We examined two types of LLM architectures, including encoder-only GatorTron and decoder-only GatorTronGPT, and evaluated their performance for the extraction of social determinants of health (SDoH) using a cross-institution dataset from the 2022 n2c2 challenge and a cross-disease dataset from the University of Florida (UF) Health. The results show that decoder-only LLMs with prompt tuning achieved better performance in cross-domain applications. GatorTronGPT achieved the best F1 scores for both datasets, outperforming traditional fine-tuned GatorTron by 8.9% and 21.8% in a cross-institution setting, and 5.5% and 14.5% in a cross-disease setting.

Related papers

• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• A Study of Large Language Models for Patient Information Extraction: Model Architecture, Fine-Tuning Strategy, and Multi-task Instruction Tuning [10.007149687726773]
  The rapid development of large language models (LLMs) has revolutionized many NLP tasks in the clinical domain.<n>This study focuses on LLM architectures, fine-tuning strategies, and multi-task instruction tuning techniques for developing robust and generalizable patient information extraction systems.
  arXiv  Detail & Related papers  (2025-09-05T02:07:40Z)
• TFOC-Net: A Short-time Fourier Transform-based Deep Learning Approach for Enhancing Cross-Subject Motor Imagery Classification [0.47498241053872914]
  Cross-subject motor imagery (CS-MI) classification in brain-computers (BCIs) is a challenging task due to the significant variability in Electroencephalography (EEG) patterns across different individuals.<n>This variability often results in lower classification accuracy compared to subject-specific models.<n>We introduce a novel approach that significantly enhances cross-subject MI classification performance through optimized preprocessing and deep learning techniques.
  arXiv  Detail & Related papers  (2025-07-03T10:17:39Z)
• RIRO: Reshaping Inputs, Refining Outputs Unlocking the Potential of Large Language Models in Data-Scarce Contexts [0.0]
  Large language models (LLMs) have significantly advanced natural language processing, excelling in areas like text generation, summarization, and question-answering. Despite their capabilities, these models face challenges when fine-tuned on small, domain-specific datasets. We introduce RIRO, a novel two-layer architecture designed to improve performance in data-scarce environments.
  arXiv  Detail & Related papers  (2024-12-15T15:48:37Z)
• An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking [50.81324768683995]
  FIRST is a novel approach that integrates a learning-to-rank objective and leveraging the logits of only the first generated token. We extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality.
  arXiv  Detail & Related papers  (2024-11-08T12:08:17Z)
• LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement [79.31084387589968]
  Pretrained large language models (LLMs) are currently state-of-the-art for solving the vast majority of natural language processing tasks. We propose LLM2LLM, a data augmentation strategy that uses a teacher LLM to enhance a small seed dataset. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular fine-tuning in the low-data regime.
  arXiv  Detail & Related papers  (2024-03-22T08:57:07Z)
• Low-resource classification of mobility functioning information in clinical sentences using large language models [0.0]
  This study evaluates the ability of publicly available large language models (LLMs) to accurately identify the presence of functioning information from clinical notes. We collect a balanced binary classification dataset of 1000 sentences from the Mobility NER dataset, which was curated from n2c2 clinical notes.
  arXiv  Detail & Related papers  (2023-12-15T20:59:17Z)
• Generative Large Language Models Are All-purpose Text Analytics Engines: Text-to-text Learning Is All Your Need [24.672621081551675]
  We formulated 7 key clinical NLP tasks as text-to-text learning and solved them using one unified generative clinical LLM. The proposed approach achieved state-of-the-art performance for 5 out of 7 major clinical NLP tasks using one unified generative LLM.
  arXiv  Detail & Related papers  (2023-12-11T04:00:26Z)
• TOP-Training: Target-Oriented Pretraining for Medical Extractive Question Answering [53.92585020805746]
  We study extractive question-answering in the medical domain (Medical-EQA) This problem has two main challenges: (i) domain specificity, and (ii) extraction-based answering style. We propose TOP-Training, a target-oriented pre-training paradigm.
  arXiv  Detail & Related papers  (2023-10-25T20:48:16Z)
• MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering [64.6741991162092]
  We present MinPrompt, a minimal data augmentation framework for open-domain question answering. We transform the raw text into a graph structure to build connections between different factual sentences. We then apply graph algorithms to identify the minimal set of sentences needed to cover the most information in the raw text. We generate QA pairs based on the identified sentence subset and train the model on the selected sentences to obtain the final model.
  arXiv  Detail & Related papers  (2023-10-08T04:44:36Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension [38.79665143111312]
  We formulate both clinical concept extraction and relation extraction using a unified prompt-based machine reading comprehension architecture. We compare our MRC models with existing deep learning models for concept extraction and end-to-end relation extraction. We evaluate the transfer learning ability of the proposed MRC models in a cross-institution setting.
  arXiv  Detail & Related papers  (2023-03-14T22:37:31Z)
• Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition [6.917786124918387]
  We compare Transformer models that are trained from scratch to fine-tuned BERT-based LLMs. We examine the impact of an additional CRF layer on such models to encourage contextual learning.
  arXiv  Detail & Related papers  (2022-10-23T16:27:31Z)
• Revisiting LSTM Networks for Semi-Supervised Text Classification via Mixed Objective Function [106.69643619725652]
  We develop a training strategy that allows even a simple BiLSTM model, when trained with cross-entropy loss, to achieve competitive results. We report state-of-the-art results for text classification task on several benchmark datasets.
  arXiv  Detail & Related papers  (2020-09-08T21:55:22Z)
• Prior Guided Feature Enrichment Network for Few-Shot Segmentation [64.91560451900125]
  State-of-the-art semantic segmentation methods require sufficient labeled data to achieve good results. Few-shot segmentation is proposed to tackle this problem by learning a model that quickly adapts to new classes with a few labeled support samples. Theses frameworks still face the challenge of generalization ability reduction on unseen classes due to inappropriate use of high-level semantic information.
  arXiv  Detail & Related papers  (2020-08-04T10:41:32Z)
• Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction [10.832917897850361]
  We studied several variants of BERT (Bidirectional Representations using Transformers) We evaluated these methods using a direct temporal relations dataset which is a semantically focused subset of the 2012 i2b2 temporal relations challenge dataset. Results: RoBERTa, which employs better pre-training strategies including using 10x larger corpus, has improved overall F measure by 0.0864 absolute score (on the 1.00 scale) and thus reducing the error rate by 24% relative to the previous state-of-the-art performance achieved with an SVM (support vector machine) model.
  arXiv  Detail & Related papers  (2020-04-13T22:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.