Common and Rare Fundus Diseases Identification Using Vision-Language Foundation Model with Knowledge of Over 400 Diseases

• URL: http://arxiv.org/abs/2406.09317v2
• Date: Sun, 30 Jun 2024 17:32:15 GMT
• Title: Common and Rare Fundus Diseases Identification Using Vision-Language Foundation Model with Knowledge of Over 400 Diseases
• Authors: Meng Wang, Tian Lin, Aidi Lin, Kai Yu, Yuanyuan Peng, Lianyu Wang, Cheng Chen, Ke Zou, Huiyu Liang, Man Chen, Xue Yao, Meiqin Zhang, Binwei Huang, Chaoxin Zheng, Peixin Zhang, Wei Chen, Yilong Luo, Yifan Chen, Honghe Xia, Tingkun Shi, Qi Zhang, Jinming Guo, Xiaolin Chen, Jingcheng Wang, Yih Chung Tham, Dianbo Liu, Wendy Wong, Sahil Thakur, Beau Fenner, Danqi Fang, Siying Liu, Qingyun Liu, Yuqiang Huang, Hongqiang Zeng, Yanda Meng, Yukun Zhou, Zehua Jiang, Minghui Qiu, Changqing Zhang, Xinjian Chen, Sophia Y Wang, Cecilia S Lee, Lucia Sobrin, Carol Y Cheung, Chi Pui Pang, Pearse A Keane, Ching-Yu Cheng, Haoyu Chen, Huazhu Fu,
• Abstract summary: Previous foundation models for retinal images were pre-trained with limited disease categories and knowledge base. To RetiZero's pre-training, we compiled 341,896 fundus images paired with text descriptions, sourced from public datasets, ophthalmic literature, and online resources. RetiZero exhibits superior performance in several downstream tasks, including zero-shot disease recognition, image-to-image retrieval, and internal- and cross-domain disease identification.
• Score: 57.27458882764811
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Previous foundation models for retinal images were pre-trained with limited disease categories and knowledge base. Here we introduce RetiZero, a vision-language foundation model that leverages knowledge from over 400 fundus diseases. To RetiZero's pre-training, we compiled 341,896 fundus images paired with text descriptions, sourced from public datasets, ophthalmic literature, and online resources, encompassing a diverse range of diseases across multiple ethnicities and countries. RetiZero exhibits superior performance in several downstream tasks, including zero-shot disease recognition, image-to-image retrieval, and internal- and cross-domain disease identification. In zero-shot scenarios, RetiZero achieves Top5 accuracy scores of 0.8430 for 15 fundus diseases and 0.7561 for 52 fundus diseases. For image retrieval, it achieves Top5 scores of 0.9500 and 0.8860 for the same disease sets, respectively. Clinical evaluations show that RetiZero's Top3 zero-shot performance surpasses the average of 19 ophthalmologists from Singapore, China and the United States. Furthermore, RetiZero significantly enhances clinicians' accuracy in diagnosing fundus disease. These findings underscore the value of integrating the RetiZero foundation model into clinical settings, where a variety of fundus diseases are encountered.

Related papers

• RET-CLIP: A Retinal Image Foundation Model Pre-trained with Clinical Diagnostic Reports [19.915033191502328]
  The Vision-Language Foundation model is increasingly investigated in the fields of computer vision and natural language processing. To handle this issue, a CLIP-style retinal image foundation model is developed in this paper. Our foundation model, RET-CLIP, is specifically trained on a dataset of 193,865 patients to extract general features of color fundus photographs.
  arXiv  Detail & Related papers  (2024-05-23T03:20:51Z)
• Uncertainty-inspired Open Set Learning for Retinal Anomaly Identification [71.06194656633447]
  We establish an uncertainty-inspired open-set (UIOS) model, which was trained with fundus images of 9 retinal conditions. Our UIOS model with thresholding strategy achieved an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set. UIOS correctly predicted high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images.
  arXiv  Detail & Related papers  (2023-04-08T10:47:41Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [51.14431540035141]
  We propose a new fine-tuning strategy based on sentence sampling and positive-pair loss relaxation for improving the downstream zero-shot pathology classification performance. Our method consistently showed dramatically improved zero-shot pathology classification performance on four different chest X-ray datasets.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)
• Detection of multiple retinal diseases in ultra-widefield fundus images using deep learning: data-driven identification of relevant regions [2.20200533591633]
  Ultra-widefield (UWF) imaging is a promising modality that captures a larger retinal field of view. Previous studies showed that deep learning (DL) models are effective for detecting retinal disease in UWF images. We propose a DL model that can recognise multiple retinal diseases under more realistic conditions.
  arXiv  Detail & Related papers  (2022-03-11T17:33:33Z)
• CD&S Dataset: Handheld Imagery Dataset Acquired Under Field Conditions for Corn Disease Identification and Severity Estimation [5.949779668853555]
  The Corn Disease and Severity dataset consisted of 4455 total images comprising of 2112 field images and 2343 augmented images. For training disease identification models, half of the imagery data for each disease was annotated using bounding boxes. For severity estimation, an additional 515 raw images for NLS were acquired and categorized into severity classes ranging from 1 (resistant) to 5 (susceptible)
  arXiv  Detail & Related papers  (2021-10-22T22:33:51Z)
• The Report on China-Spain Joint Clinical Testing for Rapid COVID-19 Risk Screening by Eye-region Manifestations [59.48245489413308]
  We developed and tested a COVID-19 rapid prescreening model using the eye-region images captured in China and Spain with cellphone cameras. The performance was measured using area under receiver-operating-characteristic curve (AUC), sensitivity, specificity, accuracy, and F1.
  arXiv  Detail & Related papers  (2021-09-18T02:28:01Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• A Benchmark of Ocular Disease Intelligent Recognition: One Shot for Multi-disease Detection [9.059366200759722]
  In ophthalmology, early fundus screening is an economic and effective way to prevent blindness caused by ophthalmic diseases. We release a dataset with 8 diseases to meet the real medical scene, which contains 10,000 fundus images from both eyes of 5,000 patients.
  arXiv  Detail & Related papers  (2021-02-16T07:00:49Z)
• Pathological myopia classification with simultaneous lesion segmentation using deep learning [16.456009188497823]
  Investigation reports on the results of convolutional neural networks developed for the recently introduced PathologicAL Myopia dataset. We propose a new Optic Nerve Head (ONH)-based prediction enhancement for the segmentation of atrophy and fovea.
  arXiv  Detail & Related papers  (2020-06-04T12:21:06Z)
• Modeling and Enhancing Low-quality Retinal Fundus Images [167.02325845822276]
  Low-quality fundus images increase uncertainty in clinical observation and lead to the risk of misdiagnosis. We propose a clinically oriented fundus enhancement network (cofe-Net) to suppress global degradation factors. Experiments on both synthetic and real images demonstrate that our algorithm effectively corrects low-quality fundus images without losing retinal details.
  arXiv  Detail & Related papers  (2020-05-12T08:01:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.