Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis

• URL: http://arxiv.org/abs/2310.09909v3
• Date: Mon, 4 Dec 2023 14:13:35 GMT
• Title: Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosis
• Authors: Chaoyi Wu, Jiayu Lei, Qiaoyu Zheng, Weike Zhao, Weixiong Lin, Xiaoman Zhang, Xiao Zhou, Ziheng Zhao, Ya Zhang, Yanfeng Wang and Weidi Xie
• Abstract summary: GPT-4V is OpenAI's newest model for multimodal medical diagnosis. Our evaluation encompasses 17 human body systems. GPT-4V demonstrates proficiency in distinguishing between medical image modalities and anatomy. It faces significant challenges in disease diagnosis and generating comprehensive reports.
• Score: 59.35504779947686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Driven by the large foundation models, the development of artificial intelligence has witnessed tremendous progress lately, leading to a surge of general interest from the public. In this study, we aim to assess the performance of OpenAI's newest model, GPT-4V(ision), specifically in the realm of multimodal medical diagnosis. Our evaluation encompasses 17 human body systems, including Central Nervous System, Head and Neck, Cardiac, Chest, Hematology, Hepatobiliary, Gastrointestinal, Urogenital, Gynecology, Obstetrics, Breast, Musculoskeletal, Spine, Vascular, Oncology, Trauma, Pediatrics, with images taken from 8 modalities used in daily clinic routine, e.g., X-ray, Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), Digital Subtraction Angiography (DSA), Mammography, Ultrasound, and Pathology. We probe the GPT-4V's ability on multiple clinical tasks with or without patent history provided, including imaging modality and anatomy recognition, disease diagnosis, report generation, disease localisation. Our observation shows that, while GPT-4V demonstrates proficiency in distinguishing between medical image modalities and anatomy, it faces significant challenges in disease diagnosis and generating comprehensive reports. These findings underscore that while large multimodal models have made significant advancements in computer vision and natural language processing, it remains far from being used to effectively support real-world medical applications and clinical decision-making. All images used in this report can be found in https://github.com/chaoyi-wu/GPT-4V_Medical_Evaluation.

Related papers

• Exploring the Feasibility of Multimodal Chatbot AI as Copilot in Pathology Diagnostics: Generalist Model's Pitfall [17.9731336178034]
  ChatGPT and other multimodal models have shown promise in transforming medical image analysis through capabilities such as medical vision-language question answering. This study benchmarks the performance of GPT on pathology images, assessing their diagnostic accuracy and efficiency in real-word clinical records. We observe significant deficits of GPT in bone diseases and a fair-level performance in diseases from other three systems.
  arXiv  Detail & Related papers  (2024-09-04T01:30:05Z)
• Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports [51.45762396192655]
  Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence for computer vision. This study evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets.
  arXiv  Detail & Related papers  (2024-07-08T09:08:42Z)
• MiniGPT-Med: Large Language Model as a General Interface for Radiology Diagnosis [28.421857904824627]
  MiniGPT-Med is a vision-language model derived from large-scale language models and tailored for medical applications. It is capable of performing tasks such as medical report generation, visual question answering (VQA), and disease identification within medical imagery. It achieves state-of-the-art performance on medical report generation, higher than the previous best model by 19% accuracy.
  arXiv  Detail & Related papers  (2024-07-04T18:21:10Z)
• GPT-4V(ision) Unsuitable for Clinical Care and Education: A Clinician-Evaluated Assessment [6.321623278767821]
  GPT-4V was recently developed for general image interpretation. Board-certified physicians and senior residents assessed GPT-4V's proficiency across a range of medical conditions. GPT-4V's diagnostic accuracy and clinical decision-making abilities are poor, posing risks to patient safety.
  arXiv  Detail & Related papers  (2023-11-14T17:06:09Z)
• Holistic Evaluation of GPT-4V for Biomedical Imaging [113.46226609088194]
  GPT-4V represents a breakthrough in artificial general intelligence for computer vision. We assess GPT-4V's performance across 16 medical imaging categories, including radiology, oncology, ophthalmology, pathology, and more. Results show GPT-4V's proficiency in modality and anatomy recognition but difficulty with disease diagnosis and localization.
  arXiv  Detail & Related papers  (2023-11-10T18:40:44Z)
• BMAD: Benchmarks for Medical Anomaly Detection [51.22159321912891]
  Anomaly detection (AD) is a fundamental research problem in machine learning and computer vision. In medical imaging, AD is especially vital for detecting and diagnosing anomalies that may indicate rare diseases or conditions. We introduce a comprehensive evaluation benchmark for assessing anomaly detection methods on medical images.
  arXiv  Detail & Related papers  (2023-06-20T20:23:46Z)
• Learned super resolution ultrasound for improved breast lesion characterization [52.77024349608834]
  Super resolution ultrasound localization microscopy enables imaging of the microvasculature at the capillary level. In this work we use a deep neural network architecture that makes effective use of signal structure to address these challenges. By leveraging our trained network, the microvasculature structure is recovered in a short time, without prior PSF knowledge, and without requiring separability of the UCAs.
  arXiv  Detail & Related papers  (2021-07-12T09:04:20Z)
• Review of Artificial Intelligence Techniques in Imaging Data Acquisition, Segmentation and Diagnosis for COVID-19 [71.41929762209328]
  The pandemic of coronavirus disease 2019 (COVID-19) is spreading all over the world. Medical imaging such as X-ray and computed tomography (CT) plays an essential role in the global fight against COVID-19. The recently emerging artificial intelligence (AI) technologies further strengthen the power of the imaging tools and help medical specialists.
  arXiv  Detail & Related papers  (2020-04-06T15:21:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.