From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPT

• URL: http://arxiv.org/abs/2502.14714v1
• Date: Thu, 20 Feb 2025 16:39:57 GMT
• Title: From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPT
• Authors: Ahmed Abdeen Hamed, Byung Suk Lee,
• Abstract summary: We present a computational approach that systematically evaluates the factual accuracy of biomedical knowledge that an LLM model has been prompted to generate. Our approach encompasses two processes: the generation of disease-centric associations and the verification of them using the semantic knowledge of the biomedical knowledge.
• Score: 3.353249333130154
• License:
• Abstract: The generative capabilities of LLM models present opportunities in accelerating tasks and concerns with the authenticity of the knowledge it produces. To address the concerns, we present a computational approach that systematically evaluates the factual accuracy of biomedical knowledge that an LLM model has been prompted to generate. Our approach encompasses two processes: the generation of disease-centric associations and the verification of them using the semantic knowledge of the biomedical ontologies. Using ChatGPT as the select LLM model, we designed a set of prompt-engineering processes to generate linkages between diseases, drugs, symptoms, and genes to establish grounds for assessments. Experimental results demonstrate high accuracy in identifying disease terms (88%-97%), drug names (90%-91%), and genetic information (88%-98%). The symptom term identification accuracy was notably lower (49%-61%), as verified against the DOID, ChEBI, SYMPTOM, and GO ontologies accordingly. The verification of associations reveals literature coverage rates of (89%-91%) among disease-drug and disease-gene associations. The low identification accuracy for symptom terms also contributed to the verification of symptom-related associations (49%-62%).

Related papers

• Survey and Improvement Strategies for Gene Prioritization with Large Language Models [61.24568051916653]
  Large language models (LLMs) have performed well in medical exams, but their effectiveness in diagnosing rare genetic diseases has not been assessed. We used multi-agent and Human Phenotype Ontology (HPO) classification to categorized patients based on phenotypes and solvability levels. At baseline, GPT-4 outperformed other LLMs, achieving near 30% accuracy in ranking causal genes correctly.
  arXiv  Detail & Related papers  (2025-01-30T23:03:03Z)
• A Knowledge-enhanced Pathology Vision-language Foundation Model for Cancer Diagnosis [58.85247337449624]
  We propose a knowledge-enhanced vision-language pre-training approach that integrates disease knowledge into the alignment within hierarchical semantic groups. KEEP achieves state-of-the-art performance in zero-shot cancer diagnostic tasks.
  arXiv  Detail & Related papers  (2024-12-17T17:45:21Z)
• Zebra-Llama: A Context-Aware Large Language Model for Democratizing Rare Disease Knowledge [0.0]
  We present Zebra-Llama, a specialized context-aware language model with high precision Retrieval Augmented Generation (RAG) capability. We focus on Ehlers-Danlos Syndrome (EDS) as our case study. EDS, affecting 1 in 5,000 individuals, exemplifies the complexities of rare diseases. Zebra-Llama demonstrates unprecedented capabilities in handling EDS-related queries.
  arXiv  Detail & Related papers  (2024-11-04T22:45:52Z)
• Assessing and Enhancing Large Language Models in Rare Disease Question-answering [64.32570472692187]
  We introduce a rare disease question-answering (ReDis-QA) dataset to evaluate the performance of Large Language Models (LLMs) in diagnosing rare diseases. We collected 1360 high-quality question-answer pairs within the ReDis-QA dataset, covering 205 rare diseases. We then benchmarked several open-source LLMs, revealing that diagnosing rare diseases remains a significant challenge for these models. Experiment results demonstrate that ReCOP can effectively improve the accuracy of LLMs on the ReDis-QA dataset by an average of 8%.
  arXiv  Detail & Related papers  (2024-08-15T21:09:09Z)
• MMIL: A novel algorithm for disease associated cell type discovery [58.044870442206914]
  Single-cell datasets often lack individual cell labels, making it challenging to identify cells associated with disease. We introduce Mixture Modeling for Multiple Learning Instance (MMIL), an expectation method that enables the training and calibration of cell-level classifiers.
  arXiv  Detail & Related papers  (2024-06-12T15:22:56Z)
• A Hybrid Machine Learning Model for Classifying Gene Mutations in Cancer using LSTM, BiLSTM, CNN, GRU, and GloVe [0.0]
  We introduce a novel hybrid ensemble model that synergistically combines LSTM, BiLSTM, CNN, GRU, and GloVe embeddings for the classification of gene mutations in cancer. Our approach achieved a training accuracy of 80.6%, precision of 81.6%, recall of 80.6%, and an F1 score of 83.1%, alongside a significantly reduced Mean Squared Error (MSE) of 2.596.
  arXiv  Detail & Related papers  (2023-07-24T21:01:46Z)
• Automated Huntington's Disease Prognosis via Biomedical Signals and Shallow Machine Learning [0.0]
  We used a premade, certified dataset collected at a clinic with 27 HD positive patients, 36 controls, and 6 unknowns with electroencephalography, electrocardiography, and functional near-infrared spectroscopy data. We found the highest accuracy was achieved by a scaled-out Extremely Randomized Trees algorithm, with area under the curve of the receiver operator characteristic of 0.963 and accuracy of 91.353%.
  arXiv  Detail & Related papers  (2023-02-04T02:20:46Z)
• Cancer Gene Profiling through Unsupervised Discovery [49.28556294619424]
  We introduce a novel, automatic and unsupervised framework to discover low-dimensional gene biomarkers. Our method is based on the LP-Stability algorithm, a high dimensional center-based unsupervised clustering algorithm. Our signature reports promising results on distinguishing immune inflammatory and immune desert tumors.
  arXiv  Detail & Related papers  (2021-02-11T09:04:45Z)
• Leaf Image-based Plant Disease Identification using Color and Texture Features [0.1657441317977376]
  The accuracy on a self-collected dataset is 82.47% for disease identification and 91.40% for healthy and diseased classification. This prototype system can be extended by adding more disease categories or targeting specific crop or disease categories.
  arXiv  Detail & Related papers  (2021-02-08T20:32:56Z)
• Relation-weighted Link Prediction for Disease Gene Identification [0.3078691410268859]
  We propose a novel machine learning method that identifies disease genes on such graphs. We show that our algorithms outperform its closest state-of-the-art competitor in disease gene identification by 24.1%. We also show that we achieve higher precision than Open Targets, the leading initiative for target identification, with respect to predicting drug targets in clinical trials for Parkinson's disease.
  arXiv  Detail & Related papers  (2020-11-10T15:09:33Z)
• Select-ProtoNet: Learning to Select for Few-Shot Disease Subtype Prediction [55.94378672172967]
  We focus on few-shot disease subtype prediction problem, identifying subgroups of similar patients. We introduce meta learning techniques to develop a new model, which can extract the common experience or knowledge from interrelated clinical tasks. Our new model is built upon a carefully designed meta-learner, called Prototypical Network, that is a simple yet effective meta learning machine for few-shot image classification.
  arXiv  Detail & Related papers  (2020-09-02T02:50:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.