MpoxVLM: A Vision-Language Model for Diagnosing Skin Lesions from Mpox Virus Infection
- URL: http://arxiv.org/abs/2411.10888v1
- Date: Sat, 16 Nov 2024 21:09:04 GMT
- Title: MpoxVLM: A Vision-Language Model for Diagnosing Skin Lesions from Mpox Virus Infection
- Authors: Xu Cao, Wenqian Ye, Kenny Moise, Megan Coffee,
- Abstract summary: Mpox (caused by the monkeypox virus) is a notable example of a zoonotic infection that often goes undiagnosed.
We propose MpoxVLM, a vision-language model (VLM) designed to detect mpox by analyzing both skin lesion images and patient clinical information.
Our work achieves 90.38% accuracy for mpox detection, offering a promising pathway to improve early diagnostic accuracy in combating mpox.
- Score: 5.161537754987037
- License:
- Abstract: In the aftermath of the COVID-19 pandemic and amid accelerating climate change, emerging infectious diseases, particularly those arising from zoonotic spillover, remain a global threat. Mpox (caused by the monkeypox virus) is a notable example of a zoonotic infection that often goes undiagnosed, especially as its rash progresses through stages, complicating detection across diverse populations with different presentations. In August 2024, the WHO Director-General declared the mpox outbreak a public health emergency of international concern for a second time. Despite the deployment of deep learning techniques for detecting diseases from skin lesion images, a robust and publicly accessible foundation model for mpox diagnosis is still lacking due to the unavailability of open-source mpox skin lesion images, multimodal clinical data, and specialized training pipelines. To address this gap, we propose MpoxVLM, a vision-language model (VLM) designed to detect mpox by analyzing both skin lesion images and patient clinical information. MpoxVLM integrates the CLIP visual encoder, an enhanced Vision Transformer (ViT) classifier for skin lesions, and LLaMA-2-7B models, pre-trained and fine-tuned on visual instruction-following question-answer pairs from our newly released mpox skin lesion dataset. Our work achieves 90.38% accuracy for mpox detection, offering a promising pathway to improve early diagnostic accuracy in combating mpox.
Related papers
- Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports [51.45762396192655]
Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence for computer vision.
This study evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets.
arXiv Detail & Related papers (2024-07-08T09:08:42Z) - A Recent Survey of the Advancements in Deep Learning Techniques for
Monkeypox Disease Detection [0.7179624965454197]
Monkeypox (MPox) is a zoonotic infectious disease induced by the MPox Virus.
This survey paper provides an extensive analysis of deep learning (DL) methods for the automatic detection of MPox in skin lesion images.
arXiv Detail & Related papers (2023-11-06T18:18:42Z) - A Web-based Mpox Skin Lesion Detection System Using State-of-the-art
Deep Learning Models Considering Racial Diversity [1.846958522363092]
'Mpox', formerly known as 'Monkeypox', has become a significant public health concern and has spread to over 110 countries globally.
Computer-aided screening tools have been proven valuable in cases where Polymerase Chain Reaction (PCR) based diagnosis is not immediately available.
Deep learning methods are powerful in learning complex data representations, but their efficacy largely depends on adequate training data.
arXiv Detail & Related papers (2023-06-25T08:23:44Z) - Classification of Human Monkeypox Disease Using Deep Learning Models and
Attention Mechanisms [0.9257985820122999]
Human monkeypox disease syndromes are quite similar to chickenpox, and measles classic symptoms.
Various deep-learning methods have shown promising performances in the image-based diagnosis of COVID-19.
An architecture consisting of Xception-CBAM-Dense layers performed better than the other models at classifying human monkeypox.
arXiv Detail & Related papers (2022-11-21T13:30:34Z) - Transfer learning and Local interpretable model agnostic based visual
approach in Monkeypox Disease Detection and Classification: A Deep Learning
insights [0.0]
The recent development of Monkeypox disease poses a global pandemic threat when the world is still fighting Coronavirus Disease 2019 (COVID-19).
We have conducted two studies where we modified and tested six distinct deep learning models-VGG16, InceptionResNetV2, ResNet50, ResNet101, MobileNetV2, and VGG19-using transfer learning approaches.
Our preliminary computational results show that the proposed modified InceptionResNetV2 and MobileNetV2 models perform best by achieving an accuracy ranging from 93% to 99%.
arXiv Detail & Related papers (2022-11-01T18:07:34Z) - Image Data collection and implementation of deep learning-based model in
detecting Monkeypox disease using modified VGG16 [0.0]
We introduce a newly developed "Monkeypox2022" dataset that is publicly available to use and can be obtained from our shared GitHub repository.
We propose and evaluate a modified VGG16 model, which includes two distinct studies: Study One and Two.
Our exploratory computational results indicate that our suggested model can identify Monkeypox patients with an accuracy of $97pm1.8%$.
arXiv Detail & Related papers (2022-06-04T00:34:15Z) - COVID-Net Biochem: An Explainability-driven Framework to Building
Machine Learning Models for Predicting Survival and Kidney Injury of COVID-19
Patients from Clinical and Biochemistry Data [66.43957431843324]
We introduce COVID-Net Biochem, a versatile and explainable framework for constructing machine learning models.
We apply this framework to predict COVID-19 patient survival and the likelihood of developing Acute Kidney Injury during hospitalization.
arXiv Detail & Related papers (2022-04-24T07:38:37Z) - COVID-19 Detection from Chest X-ray Images using Imprinted Weights
Approach [67.05664774727208]
Chest radiography is an alternative screening method for the COVID-19.
Computer-aided diagnosis (CAD) has proven to be a viable solution at low cost and with fast speed.
To address this challenge, we propose the use of a low-shot learning approach named imprinted weights.
arXiv Detail & Related papers (2021-05-04T19:01:40Z) - COVIDx-US -- An open-access benchmark dataset of ultrasound imaging data
for AI-driven COVID-19 analytics [116.6248556979572]
COVIDx-US is an open-access benchmark dataset of COVID-19 related ultrasound imaging data.
It consists of 93 lung ultrasound videos and 10,774 processed images of patients infected with SARS-CoV-2 pneumonia, non-SARS-CoV-2 pneumonia, as well as healthy control cases.
arXiv Detail & Related papers (2021-03-18T03:31:33Z) - Cross-lingual Transfer Learning for COVID-19 Outbreak Alignment [90.12602012910465]
We train on Italy's early COVID-19 outbreak through Twitter and transfer to several other countries.
Our experiments show strong results with up to 0.85 Spearman correlation in cross-country predictions.
arXiv Detail & Related papers (2020-06-05T02:04:25Z) - Viral Pneumonia Screening on Chest X-ray Images Using Confidence-Aware
Anomaly Detection [86.81773672627406]
Clusters of viral pneumonia during a short period of time may be a harbinger of an outbreak or pandemic, like SARS, MERS, and recent COVID-19.
Rapid and accurate detection of viral pneumonia using chest X-ray can be significantly useful in large-scale screening and epidemic prevention.
Viral pneumonia often have diverse causes and exhibit notably different visual appearances on X-ray images.
arXiv Detail & Related papers (2020-03-27T11:32:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.