MultiRetNet: A Multimodal Vision Model and Deferral System for Staging Diabetic Retinopathy

• URL: http://arxiv.org/abs/2507.14738v1
• Date: Sat, 19 Jul 2025 20:00:31 GMT
• Title: MultiRetNet: A Multimodal Vision Model and Deferral System for Staging Diabetic Retinopathy
• Authors: Jeannie She, Katie Spivakovsky,
• Abstract summary: Diabetic retinopathy (DR) is a leading cause of preventable blindness, affecting over 100 million people worldwide.<n>Individuals from lower-income communities face a higher risk of progressing to advanced stages before diagnosis, largely due to limited access to screening.<n>We propose MultiRetNet, a novel pipeline combining retinal imaging, socioeconomic factors, and comorbidity profiles to improve DR staging accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diabetic retinopathy (DR) is a leading cause of preventable blindness, affecting over 100 million people worldwide. In the United States, individuals from lower-income communities face a higher risk of progressing to advanced stages before diagnosis, largely due to limited access to screening. Comorbid conditions further accelerate disease progression. We propose MultiRetNet, a novel pipeline combining retinal imaging, socioeconomic factors, and comorbidity profiles to improve DR staging accuracy, integrated with a clinical deferral system for a clinical human-in-the-loop implementation. We experiment with three multimodal fusion methods and identify fusion through a fully connected layer as the most versatile methodology. We synthesize adversarial, low-quality images and use contrastive learning to train the deferral system, guiding the model to identify out-of-distribution samples that warrant clinician review. By maintaining diagnostic accuracy on suboptimal images and integrating critical health data, our system can improve early detection, particularly in underserved populations where advanced DR is often first identified. This approach may reduce healthcare costs, increase early detection rates, and address disparities in access to care, promoting healthcare equity.

Related papers

• A Novel Multimodal Framework for Early Detection of Alzheimers Disease Using Deep Learning [0.0]
  Alzheimers Disease (AD) is a progressive neurodegenerative disorder that poses significant challenges in its early diagnosis.<n>Traditional diagnostic methods fall short of capturing the multifaceted nature of the disease.<n>We propose a novel framework for the early detection of AD that integrates data from three primary sources: MRI imaging, cognitive assessments, and biomarkers.
  arXiv  Detail & Related papers  (2025-08-05T03:46:59Z)
• The Role of AI in Early Detection of Life-Threatening Diseases: A Retinal Imaging Perspective [10.884863227198975]
  We systematically synthesize the latest OCT/A and AO developments, AI/ML approaches, and mHealth/Tele-ophthalmology initiatives.<n>We propose a roadmap for multicenter protocol standardization, prospective validation trials, and seamless incorporation of retinal screening into primary and specialty care pathways.
  arXiv  Detail & Related papers  (2025-05-27T07:19:37Z)
• Retinal Fundus Multi-Disease Image Classification using Hybrid CNN-Transformer-Ensemble Architectures [0.3277163122167434]
  Our research is motivated by the urgent global issue of a large population affected by retinal diseases.<n>Our primary objective is to develop a comprehensive diagnostic system capable of accurately predicting retinal diseases.
  arXiv  Detail & Related papers  (2025-03-27T12:55:07Z)
• Continually Evolved Multimodal Foundation Models for Cancer Prognosis [50.43145292874533]
  Cancer prognosis is a critical task that involves predicting patient outcomes and survival rates.<n>Previous studies have integrated diverse data modalities, such as clinical notes, medical images, and genomic data, leveraging their complementary information.<n>Existing approaches face two major limitations. First, they struggle to incorporate newly arrived data with varying distributions into training, such as patient records from different hospitals.<n>Second, most multimodal integration methods rely on simplistic concatenation or task-specific pipelines, which fail to capture the complex interdependencies across modalities.
  arXiv  Detail & Related papers  (2025-01-30T06:49:57Z)
• Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration [54.76511683427566]
  This research introduces a novel multimodal method for classifying skin lesions, integrating smartphone-captured images with essential clinical and demographic information. A distinctive aspect of this method is the integration of an auxiliary task focused on super-resolution image prediction. The experimental evaluations have been conducted using the PAD-UFES20 dataset, applying various deep-learning architectures.
  arXiv  Detail & Related papers  (2024-02-16T05:16:20Z)
• Diagnosing Alzheimer's Disease using Early-Late Multimodal Data Fusion with Jacobian Maps [1.5501208213584152]
  Alzheimer's disease (AD) is a prevalent and debilitating neurodegenerative disorder impacting a large aging population. We propose an efficient early-late fusion (ELF) approach, which leverages a convolutional neural network for automated feature extraction and random forests. To tackle the challenge of detecting subtle changes in brain volume, we transform images into the Jacobian domain (JD)
  arXiv  Detail & Related papers  (2023-10-25T19:02:57Z)
• Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images [1.6449510885987357]
  Diabetic Retinopathy (DR), a prevalent and severe complication of diabetes, affects millions of individuals globally. Recent advancements in imaging technologies provide opportunities for the early detection of DR but also pose significant challenges. This study introduces a novel multimodal approach that leverages these imaging modalities to notably enhance DR classification.
  arXiv  Detail & Related papers  (2023-10-03T09:35:38Z)
• Automatic diagnosis of knee osteoarthritis severity using Swin transformer [55.01037422579516]
  Knee osteoarthritis (KOA) is a widespread condition that can cause chronic pain and stiffness in the knee joint. We propose an automated approach that employs the Swin Transformer to predict the severity of KOA.
  arXiv  Detail & Related papers  (2023-07-10T09:49:30Z)
• Robust and Efficient Medical Imaging with Self-Supervision [80.62711706785834]
  We present REMEDIS, a unified representation learning strategy to improve robustness and data-efficiency of medical imaging AI. We study a diverse range of medical imaging tasks and simulate three realistic application scenarios using retrospective data.
  arXiv  Detail & Related papers  (2022-05-19T17:34:18Z)
• An Interpretable Multiple-Instance Approach for the Detection of referable Diabetic Retinopathy from Fundus Images [72.94446225783697]
  We propose a machine learning system for the detection of referable Diabetic Retinopathy in fundus images. By extracting local information from image patches and combining it efficiently through an attention mechanism, our system is able to achieve high classification accuracy. We evaluate our approach on publicly available retinal image datasets, in which it exhibits near state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-03-02T13:14:15Z)
• Inheritance-guided Hierarchical Assignment for Clinical Automatic Diagnosis [50.15205065710629]
  Clinical diagnosis, which aims to assign diagnosis codes for a patient based on the clinical note, plays an essential role in clinical decision-making. We propose a novel framework to combine the inheritance-guided hierarchical assignment and co-occurrence graph propagation for clinical automatic diagnosis.
  arXiv  Detail & Related papers  (2021-01-27T13:16:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.