Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing

• URL: http://arxiv.org/abs/2406.06723v1
• Date: Mon, 10 Jun 2024 18:34:48 GMT
• Title: Leveraging Large Language Models for Knowledge-free Weak Supervision in Clinical Natural Language Processing
• Authors: Enshuo Hsu, Kirk Roberts,
• Abstract summary: We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7% to 47.9% in F1 scores.
• Score: 3.7906296809297406
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The performance of deep learning-based natural language processing systems is based on large amounts of labeled training data which, in the clinical domain, are not easily available or affordable. Weak supervision and in-context learning offer partial solutions to this issue, particularly using large language models (LLMs), but their performance still trails traditional supervised methods with moderate amounts of gold-standard data. In particular, inferencing with LLMs is computationally heavy. We propose an approach leveraging fine-tuning LLMs and weak supervision with virtually no domain knowledge that still achieves consistently dominant performance. Using a prompt-based approach, the LLM is used to generate weakly-labeled data for training a downstream BERT model. The weakly supervised model is then further fine-tuned on small amounts of gold standard data. We evaluate this approach using Llama2 on three different n2c2 datasets. With no more than 10 gold standard notes, our final BERT models weakly supervised by fine-tuned Llama2-13B consistently outperformed out-of-the-box PubMedBERT by 4.7% to 47.9% in F1 scores. With only 50 gold standard notes, our models achieved close performance to fully fine-tuned systems.

Related papers

• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference. Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress. LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset. Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Evaluating Large Language Models for Health-Related Text Classification Tasks with Public Social Media Data [3.9459077974367833]
  Large language models (LLMs) have demonstrated remarkable success in NLP tasks. We benchmarked one supervised classic machine learning model based on Support Vector Machines (SVMs), three supervised pretrained language models (PLMs) based on RoBERTa, BERTweet, and SocBERT, and two LLM based classifiers (GPT3.5 and GPT4), across 6 text classification tasks. Our comprehensive experiments demonstrate that employ-ing data augmentation using LLMs (GPT-4) with relatively small human-annotated data to train lightweight supervised classification models achieves superior results compared to training with human-annotated data
  arXiv  Detail & Related papers  (2024-03-27T22:05:10Z)
• Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning [50.9692060692705]
  This paper introduces $textbfLanguage Models for $textbfMo$tion Control ($textbfLaMo$), a general framework based on Decision Transformers for offline RL. Our framework highlights four crucial components:. Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method,. In particular, our method demonstrates superior performance in scenarios with limited data samples.
  arXiv  Detail & Related papers  (2023-10-31T16:24:17Z)
• LLMaAA: Making Large Language Models as Active Annotators [32.57011151031332]
  We propose LLMaAA, which takes large language models as annotators and puts them into an active learning loop to determine what to annotate efficiently. We conduct experiments and analysis on two classic NLP tasks, named entity recognition and relation extraction. With LLMaAA, task-specific models trained from LLM-generated labels can outperform the teacher within only hundreds of annotated examples.
  arXiv  Detail & Related papers  (2023-10-30T14:54:15Z)
• Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding [38.11411155621616]
  We study self-training as one of the predominant semi-supervised learning approaches. We present UPET, a novel Uncertainty-aware self-Training framework. We show that UPET achieves a substantial improvement in terms of performance and efficiency.
  arXiv  Detail & Related papers  (2023-10-19T02:18:29Z)
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes [91.58845026796149]
  We introduce Distilling step-by-step, a new mechanism that trains small models that outperform large language models. We present three findings across 4 NLP benchmarks.
  arXiv  Detail & Related papers  (2023-05-03T17:50:56Z)
• Clinical Prompt Learning with Frozen Language Models [4.077071350659386]
  Large but frozen pre-trained language models (PLMs) with prompt learning outperform smaller but fine-tuned models. We investigated the viability of prompt learning on clinically meaningful decision tasks. Results are partially in line with the prompt learning literature, with prompt learning able to match or improve on traditional fine-tuning.
  arXiv  Detail & Related papers  (2022-05-11T14:25:13Z)
• Bayesian Active Learning with Pretrained Language Models [9.161353418331245]
  Active Learning (AL) is a method to iteratively select data for annotation from a pool of unlabeled data. Previous AL approaches have been limited to task-specific models that are trained from scratch at each iteration. We introduce BALM; Bayesian Active Learning with pretrained language models.
  arXiv  Detail & Related papers  (2021-04-16T19:07:31Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.