Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study

• URL: http://arxiv.org/abs/2402.15663v1
• Date: Sat, 24 Feb 2024 00:38:29 GMT
• Title: Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study
• Authors: Zhaoyue Sun, Gabriele Pergola, Byron C. Wallace and Yulan He
• Abstract summary: This research aims to investigate the ability of large language models, specifically ChatGPT, in the context of pharmacovigilance event extraction. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task. The inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels.
• Score: 38.555547784219115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the context of pharmacovigilance event extraction, of which the main goal is to identify and extract adverse events or potential therapeutic events from textual medical sources. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task, employing various prompts and demonstration selection strategies. The findings demonstrate that while ChatGPT demonstrates reasonable performance with appropriate demonstration selection strategies, it still falls short compared to fully fine-tuned small models. Additionally, we explore the potential of leveraging ChatGPT for data augmentation. However, our investigation reveals that the inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels. To mitigate this, we explore different filtering strategies and find that, with the proper approach, more stable performance can be achieved, although constant improvement remains elusive.

Related papers

• Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development [12.258245804049114]
  The mining of adverse drug events (ADEs) is pivotal in pharmacovigilance, enhancing patient safety. Traditional ADE detection methods are reliable but slow, not easily adaptable to large-scale operations. Previous ADE mining studies have focused on text-based methodologies, overlooking visual cues. We present a MultiModal Adverse Drug Event (MMADE) detection dataset, merging ADE-related textual information with visual aids.
  arXiv  Detail & Related papers  (2024-05-24T17:58:42Z)
• CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study [17.96401880059829]
  Large Language Models (LLMs) such as ChatGPT have achieved tremendous success in various downstream tasks. We propose to use a knowledge graph as auxiliary information to guide the LLMs in making predictions. Our few-shot learning method achieves satisfactory performance compared with fine-tuning strategies.
  arXiv  Detail & Related papers  (2023-07-21T04:43:00Z)
• SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design [64.69434941796904]
  We propose a novel setting and models for in-context drug synergy learning. We are given a small "personalized dataset" of 10-20 drug synergy relationships in the context of specific cancer cell targets. Our goal is to predict additional drug synergy relationships in that context.
  arXiv  Detail & Related papers  (2023-06-19T17:03:46Z)
• Pushing the Limits of ChatGPT on NLP Tasks [79.17291002710517]
  Despite the success of ChatGPT, its performances on most NLP tasks are still well below the supervised baselines. In this work, we looked into the causes, and discovered that its subpar performance was caused by the following factors. We propose a collection of general modules to address these issues, in an attempt to push the limits of ChatGPT on NLP tasks.
  arXiv  Detail & Related papers  (2023-06-16T09:40:05Z)
• To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
  This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
  arXiv  Detail & Related papers  (2023-04-04T03:04:28Z)
• Does Synthetic Data Generation of LLMs Help Clinical Text Mining? [51.205078179427645]
  We investigate the potential of OpenAI's ChatGPT to aid in clinical text mining. We propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data. Our method has resulted in significant improvements in the performance of downstream tasks.
  arXiv  Detail & Related papers  (2023-03-08T03:56:31Z)
• Containing a spread through sequential learning: to exploit or to explore? [11.86456063377268]
  We design testing and isolation strategies to contain the spread and minimize the cumulative infections under a given test budget. We show that exploration can outperform exploitation in various synthetic and real-data networks depending on the parameters of the network and the spread.
  arXiv  Detail & Related papers  (2023-03-01T00:13:52Z)
• ALLSH: Active Learning Guided by Local Sensitivity and Hardness [98.61023158378407]
  We propose to retrieve unlabeled samples with a local sensitivity and hardness-aware acquisition function. Our method achieves consistent gains over the commonly used active learning strategies in various classification tasks.
  arXiv  Detail & Related papers  (2022-05-10T15:39:11Z)
• Adding more data does not always help: A study in medical conversation summarization with PEGASUS [5.276054618115727]
  We study the effect of dataset size on transfer learning medical conversation summarization using PEG. We also evaluate various iterative labeling strategies in the low-data regime, following their success in the classification setting. Our work sheds light on the successes and challenges of translating low-data regime techniques in classification to medical conversation summarization.
  arXiv  Detail & Related papers  (2021-11-15T07:27:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.