Comparative Analysis of Drug-GPT and ChatGPT LLMs for Healthcare Insights: Evaluating Accuracy and Relevance in Patient and HCP Contexts

• URL: http://arxiv.org/abs/2307.16850v1
• Date: Mon, 24 Jul 2023 19:27:11 GMT
• Title: Comparative Analysis of Drug-GPT and ChatGPT LLMs for Healthcare Insights: Evaluating Accuracy and Relevance in Patient and HCP Contexts
• Authors: Giorgos Lysandrou, Roma English Owen, Kirsty Mursec, Grant Le Brun, Elizabeth A. L. Fairley
• Abstract summary: This study presents a comparative analysis of three Generative Pre-trained Transformer (GPT) solutions in a question and answer (Q&A) setting. The objective is to determine which model delivers the most accurate and relevant information in response to prompts related to patient experiences with atopic dermatitis (AD) and healthcare professional (HCP) discussions about diabetes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This study presents a comparative analysis of three Generative Pre-trained Transformer (GPT) solutions in a question and answer (Q&A) setting: Drug-GPT 3, Drug-GPT 4, and ChatGPT, in the context of healthcare applications. The objective is to determine which model delivers the most accurate and relevant information in response to prompts related to patient experiences with atopic dermatitis (AD) and healthcare professional (HCP) discussions about diabetes. The results demonstrate that while all three models are capable of generating relevant and accurate responses, Drug-GPT 3 and Drug-GPT 4, which are supported by curated datasets of patient and HCP social media and message board posts, provide more targeted and in-depth insights. ChatGPT, a more general-purpose model, generates broader and more general responses, which may be valuable for readers seeking a high-level understanding of the topics but may lack the depth and personal insights found in the answers generated by the specialized Drug-GPT models. This comparative analysis highlights the importance of considering the language model's perspective, depth of knowledge, and currency when evaluating the usefulness of generated information in healthcare applications.

Related papers

• MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models [3.0874677990361246]
  Large Language Models (LLMs) have shown impressive capabilities in generating human-like responses. We propose MedInsight:a novel retrieval framework that augments LLM inputs with relevant background information. Experiments on the MTSamples dataset validate MedInsight's effectiveness in generating contextually appropriate medical responses.
  arXiv  Detail & Related papers  (2024-03-13T15:20:30Z)
• Large Language Models in Medical Term Classification and Unexpected Misalignment Between Response and Reasoning [28.355000184014084]
  This study assesses the ability of state-of-the-art large language models (LLMs) to identify patients with mild cognitive impairment (MCI) from discharge summaries. The data was partitioned into training, validation, and testing sets in a 7:2:1 ratio for model fine-tuning and evaluation. Open-source models like Falcon and LLaMA 2 achieved high accuracy but lacked explanatory reasoning.
  arXiv  Detail & Related papers  (2023-12-19T17:36:48Z)
• A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering [56.01977227584777]
  multimodal large models (MLMs) has significantly advanced the field of visual understanding. Yet, the true challenge lies in the domain of knowledge-intensive visual question answering (VQA) tasks. This study provides an in-depth evaluation of the newly introduced GPT-4V.
  arXiv  Detail & Related papers  (2023-11-13T18:22:32Z)
• A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysis [87.25494411021066]
  GPT-4V's multimodal capability for medical image analysis is evaluated. It is found that GPT-4V excels in understanding medical images and generates high-quality radiology reports. It is found that its performance for medical visual grounding needs to be substantially improved.
  arXiv  Detail & Related papers  (2023-10-31T11:39:09Z)
• Clairvoyance: A Pipeline Toolkit for Medical Time Series [95.22483029602921]
  Time-series learning is the bread and butter of data-driven *clinical decision support* Clairvoyance proposes a unified, end-to-end, autoML-friendly pipeline that serves as a software toolkit. Clairvoyance is the first to demonstrate viability of a comprehensive and automatable pipeline for clinical time-series ML.
  arXiv  Detail & Related papers  (2023-10-28T12:08:03Z)
• Navigating Healthcare Insights: A Birds Eye View of Explainability with Knowledge Graphs [0.0]
  Knowledge graphs (KGs) are gaining prominence in Healthcare AI, especially in drug discovery and pharmaceutical research. This overview summarizes recent literature on the impact of KGs in healthcare and their role in developing explainable AI models. We emphasize the importance of making KGs more interpretable through knowledge-infused learning in healthcare.
  arXiv  Detail & Related papers  (2023-09-28T16:57:03Z)
• A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises [53.48844796428081]
  This work provides the first comprehensive review of healthcare knowledge graphs (HKGs) It summarizes the pipeline and key techniques for HKG construction, as well as the common utilization approaches. At the application level, we delve into the successful integration of HKGs across various health domains.
  arXiv  Detail & Related papers  (2023-06-07T21:51:56Z)
• Informing clinical assessment by contextualizing post-hoc explanations of risk prediction models in type-2 diabetes [50.8044927215346]
  We consider a comorbidity risk prediction scenario and focus on contexts regarding the patients clinical state. We employ several state-of-the-art LLMs to present contexts around risk prediction model inferences and evaluate their acceptability. Our paper is one of the first end-to-end analyses identifying the feasibility and benefits of contextual explanations in a real-world clinical use case.
  arXiv  Detail & Related papers  (2023-02-11T18:07:11Z)
• Self-supervised Answer Retrieval on Clinical Notes [68.87777592015402]
  We introduce CAPR, a rule-based self-supervision objective for training Transformer language models for domain-specific passage matching. We apply our objective in four Transformer-based architectures: Contextual Document Vectors, Bi-, Poly- and Cross-encoders. We report that CAPR outperforms strong baselines in the retrieval of domain-specific passages and effectively generalizes across rule-based and human-labeled passages.
  arXiv  Detail & Related papers  (2021-08-02T10:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.