From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPT

• URL: http://arxiv.org/abs/2502.14714v2
• Date: Sun, 23 Mar 2025 16:02:53 GMT
• Title: From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPT
• Authors: Ahmed Abdeen Hamed, Alessandro Crimi, Magdalena M. Misiak, Byung Suk Lee,
• Abstract summary: Our approach consists of two processes: generating disease-centric associations and verifying these associations.<n>Using ChatGPT as the selected LLM, we designed prompt-engineering processes to establish linkages between diseases and related drugs, symptoms, and genes.
• Score: 45.6537455491436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The generative capabilities of LLM models offer opportunities for accelerating tasks but raise concerns about the authenticity of the knowledge they produce. To address these concerns, we present a computational approach that evaluates the factual accuracy of biomedical knowledge generated by an LLM. Our approach consists of two processes: generating disease-centric associations and verifying these associations using the semantic framework of biomedical ontologies. Using ChatGPT as the selected LLM, we designed prompt-engineering processes to establish linkages between diseases and related drugs, symptoms, and genes, and assessed consistency across multiple ChatGPT models (e.g., GPT-turbo, GPT-4, etc.). Experimental results demonstrate high accuracy in identifying disease terms (88%-97%), drug names (90%-91%), and genetic information (88%-98%). However, symptom term identification was notably lower (49%-61%), due to the informal and verbose nature of symptom descriptions, which hindered effective semantic matching with the formal language of specialized ontologies. Verification of associations reveals literature coverage rates of 89%-91% for disease-drug and disease-gene pairs, while symptom-related associations exhibit lower coverage (49%-62%).

Related papers

• Survey and Improvement Strategies for Gene Prioritization with Large Language Models [61.24568051916653]
  Large language models (LLMs) have performed well in medical exams, but their effectiveness in diagnosing rare genetic diseases has not been assessed.<n>We used multi-agent and Human Phenotype Ontology (HPO) classification to categorized patients based on phenotypes and solvability levels.<n>At baseline, GPT-4 outperformed other LLMs, achieving near 30% accuracy in ranking causal genes correctly.
  arXiv  Detail & Related papers  (2025-01-30T23:03:03Z)
• A Knowledge-enhanced Pathology Vision-language Foundation Model for Cancer Diagnosis [58.85247337449624]
  We propose a knowledge-enhanced vision-language pre-training approach that integrates disease knowledge into the alignment within hierarchical semantic groups. KEEP achieves state-of-the-art performance in zero-shot cancer diagnostic tasks.
  arXiv  Detail & Related papers  (2024-12-17T17:45:21Z)
• Assessing and Enhancing Large Language Models in Rare Disease Question-answering [64.32570472692187]
  We introduce a rare disease question-answering (ReDis-QA) dataset to evaluate the performance of Large Language Models (LLMs) in diagnosing rare diseases. We collected 1360 high-quality question-answer pairs within the ReDis-QA dataset, covering 205 rare diseases. We then benchmarked several open-source LLMs, revealing that diagnosing rare diseases remains a significant challenge for these models. Experiment results demonstrate that ReCOP can effectively improve the accuracy of LLMs on the ReDis-QA dataset by an average of 8%.
  arXiv  Detail & Related papers  (2024-08-15T21:09:09Z)
• Fact-Checking Generative AI: Ontology-Driven Biological Graphs for Disease-Gene Link Verification [45.65374554914359]
  We aim to achieve fact-checking of the knowledge embedded in biological graphs that were contrived from ChatGPT contents. We adopted a biological networks approach that enables the systematic interrogation of ChatGPT's linked entities. This study demonstrated high accuracy of aggregate disease-gene links relationships found in ChatGPT-generated texts.
  arXiv  Detail & Related papers  (2023-08-07T22:13:30Z)
• A Hybrid Machine Learning Model for Classifying Gene Mutations in Cancer using LSTM, BiLSTM, CNN, GRU, and GloVe [0.0]
  We introduce a novel hybrid ensemble model that synergistically combines LSTM, BiLSTM, CNN, GRU, and GloVe embeddings for the classification of gene mutations in cancer. Our approach achieved a training accuracy of 80.6%, precision of 81.6%, recall of 80.6%, and an F1 score of 83.1%, alongside a significantly reduced Mean Squared Error (MSE) of 2.596.
  arXiv  Detail & Related papers  (2023-07-24T21:01:46Z)
• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation [0.0]
  This work explores the potential of Large Language Models for dialoguing with biomedical background knowledge. The framework involves of three evaluation steps, each assessing different aspects sequentially: fluency, prompt alignment, semantic coherence, factual knowledge, and specificity of the generated responses. The work provides a systematic assessment on the ability of eleven state-of-the-art models LLMs, including ChatGPT, GPT-4 and Llama 2, in two prompting-based tasks.
  arXiv  Detail & Related papers  (2023-05-28T22:46:21Z)
• Does Synthetic Data Generation of LLMs Help Clinical Text Mining? [51.205078179427645]
  We investigate the potential of OpenAI's ChatGPT to aid in clinical text mining. We propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data. Our method has resulted in significant improvements in the performance of downstream tasks.
  arXiv  Detail & Related papers  (2023-03-08T03:56:31Z)
• Explainable Identification of Dementia from Transcripts using Transformer Networks [0.0]
  Alzheimer's disease (AD) is the main cause of dementia which is accompanied by loss of memory and may lead to severe consequences in peoples' everyday life if not diagnosed on time. We introduce two multi-task learning models, where the main task refers to the identification of dementia (binary classification) and the auxiliary one corresponds to the identification of the severity of dementia (multiclass classification) Our model obtains accuracy equal to 84.99% on the detection of AD patients in the multi-task learning setting.
  arXiv  Detail & Related papers  (2021-09-14T21:49:05Z)
• Cancer Gene Profiling through Unsupervised Discovery [49.28556294619424]
  We introduce a novel, automatic and unsupervised framework to discover low-dimensional gene biomarkers. Our method is based on the LP-Stability algorithm, a high dimensional center-based unsupervised clustering algorithm. Our signature reports promising results on distinguishing immune inflammatory and immune desert tumors.
  arXiv  Detail & Related papers  (2021-02-11T09:04:45Z)
• Leaf Image-based Plant Disease Identification using Color and Texture Features [0.1657441317977376]
  The accuracy on a self-collected dataset is 82.47% for disease identification and 91.40% for healthy and diseased classification. This prototype system can be extended by adding more disease categories or targeting specific crop or disease categories.
  arXiv  Detail & Related papers  (2021-02-08T20:32:56Z)
• Relation-weighted Link Prediction for Disease Gene Identification [0.3078691410268859]
  We propose a novel machine learning method that identifies disease genes on such graphs. We show that our algorithms outperform its closest state-of-the-art competitor in disease gene identification by 24.1%. We also show that we achieve higher precision than Open Targets, the leading initiative for target identification, with respect to predicting drug targets in clinical trials for Parkinson's disease.
  arXiv  Detail & Related papers  (2020-11-10T15:09:33Z)
• Select-ProtoNet: Learning to Select for Few-Shot Disease Subtype Prediction [55.94378672172967]
  We focus on few-shot disease subtype prediction problem, identifying subgroups of similar patients. We introduce meta learning techniques to develop a new model, which can extract the common experience or knowledge from interrelated clinical tasks. Our new model is built upon a carefully designed meta-learner, called Prototypical Network, that is a simple yet effective meta learning machine for few-shot image classification.
  arXiv  Detail & Related papers  (2020-09-02T02:50:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.