Temporal-Enhanced Interpretable Multi-Modal Prognosis and Risk Stratification Framework for Diabetic Retinopathy (TIMM-ProRS)

• URL: http://arxiv.org/abs/2601.08240v1
• Date: Tue, 13 Jan 2026 05:57:47 GMT
• Title: Temporal-Enhanced Interpretable Multi-Modal Prognosis and Risk Stratification Framework for Diabetic Retinopathy (TIMM-ProRS)
• Abstract summary: This study introduces TIMM-ProRS, a novel deep learning framework integrating Vision Transformer (ViT), Convolutional Neural Network (CNN), and Graph Neural Network (GNN)<n>TIMM-ProRS uniquely leverages both retinal images and temporal biomarkers (HbA1c, retinal thickness) to capture multi-modal and temporal dynamics.<n>The model achieves 97.8% accuracy and an F1-score of 0.96, demonstrating state-of-the-art performance and outperforming existing methods like RSG-Net and DeepDR.
• Score: 0.17398560678845076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diabetic retinopathy (DR), affecting millions globally with projections indicating a significant rise, poses a severe blindness risk and strains healthcare systems. Diagnostic complexity arises from visual symptom overlap with conditions like age-related macular degeneration and hypertensive retinopathy, exacerbated by high misdiagnosis rates in underserved regions. This study introduces TIMM-ProRS, a novel deep learning framework integrating Vision Transformer (ViT), Convolutional Neural Network (CNN), and Graph Neural Network (GNN) with multi-modal fusion. TIMM-ProRS uniquely leverages both retinal images and temporal biomarkers (HbA1c, retinal thickness) to capture multi-modal and temporal dynamics. Evaluated comprehensively across diverse datasets including APTOS 2019 (trained), Messidor-2, RFMiD, EyePACS, and Messidor-1 (validated), the model achieves 97.8\% accuracy and an F1-score of 0.96, demonstrating state-of-the-art performance and outperforming existing methods like RSG-Net and DeepDR. This approach enables early, precise, and interpretable diagnosis, supporting scalable telemedical management and enhancing global eye health sustainability.

Related papers

• Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology [47.79180201495836]
  MixTIME is a multimodal foundation model that integrates pathology foundation models trained across distinct modalities.<n>It predicts multiplex immunofluorescence (mIF) protein expression from hematoxylin and eosin whole-slide images.<n>MixTIME achieves state-of-the-art performance across 17 protein markers as measured by correlation metrics.
  arXiv  Detail & Related papers  (2026-06-16T16:22:42Z)
• Quasi-multimodal-based pathophysiological feature learning for retinal disease diagnosis [4.437523386839875]
  A unified framework that integrates multimodal data synthesis and fusion is proposed for retinal disease classification and grading.<n>The proposed learning system is thoroughly interpreted through visualizations in both image and feature spaces.<n>This work not only enhances the accuracy and efficiency of retinal disease screening but also offers a scalable framework for data augmentation across various medical imaging modalities.
  arXiv  Detail & Related papers  (2026-02-03T15:13:57Z)
• Beyond CLIP: Knowledge-Enhanced Multimodal Transformers for Cross-Modal Alignment in Diabetic Retinopathy Diagnosis [7.945705180020063]
  We propose a knowledge-enhanced joint embedding framework that integrates retinal fundus images, clinical text, and structured patient data.<n>Our framework achieves near-perfect text-to-image retrieval performance with Recall@1 of 99.94% compared to fine-tuned CLIP's 1.29%.
  arXiv  Detail & Related papers  (2025-12-22T18:41:45Z)
• R-GenIMA: Integrating Neuroimaging and Genetics with Interpretable Multimodal AI for Alzheimer's Disease Progression [63.97617759805451]
  Early detection of Alzheimer's disease requires models capable of integrating macro-scale neuroanatomical alterations with micro-scale genetic susceptibility.<n>We introduce R-GenIMA, an interpretable multimodal large language model that couples a novel ROI-wise vision transformer with genetic prompting.<n>R-GenIMA achieves state-of-the-art performance in four-way classification across normal cognition, subjective memory concerns, mild cognitive impairment, and AD.
  arXiv  Detail & Related papers  (2025-12-22T02:54:10Z)
• DRetNet: A Novel Deep Learning Framework for Diabetic Retinopathy Diagnosis [8.234135343778993]
  Current DR detection systems struggle with poor-quality images, lack interpretability, and insufficient integration of domain-specific knowledge.<n>We introduce a novel framework that integrates three innovative contributions.<n>The framework achieves an accuracy of 92.7%, a precision of 92.5%, a recall of 92.6%, an F1-score of 92.5%, an AUC of 97.8%, a mAP of 0.96, and an MCC of 0.85.
  arXiv  Detail & Related papers  (2025-09-01T02:27:16Z)
• Design and Validation of a Responsible Artificial Intelligence-based System for the Referral of Diabetic Retinopathy Patients [65.57160385098935]
  Early detection of Diabetic Retinopathy can reduce the risk of vision loss by up to 95%.<n>We developed RAIS-DR, a Responsible AI System for DR screening that incorporates ethical principles across the AI lifecycle.<n>We evaluated RAIS-DR against the FDA-approved EyeArt system on a local dataset of 1,046 patients, unseen by both systems.
  arXiv  Detail & Related papers  (2025-08-17T21:54:11Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• MultiRetNet: A Multimodal Vision Model and Deferral System for Staging Diabetic Retinopathy [0.0]
  Diabetic retinopathy (DR) is a leading cause of preventable blindness, affecting over 100 million people worldwide.<n>Individuals from lower-income communities face a higher risk of progressing to advanced stages before diagnosis, largely due to limited access to screening.<n>We propose MultiRetNet, a novel pipeline combining retinal imaging, socioeconomic factors, and comorbidity profiles to improve DR staging accuracy.
  arXiv  Detail & Related papers  (2025-07-19T20:00:31Z)
• The Role of AI in Early Detection of Life-Threatening Diseases: A Retinal Imaging Perspective [10.884863227198975]
  We systematically synthesize the latest OCT/A and AO developments, AI/ML approaches, and mHealth/Tele-ophthalmology initiatives.<n>We propose a roadmap for multicenter protocol standardization, prospective validation trials, and seamless incorporation of retinal screening into primary and specialty care pathways.
  arXiv  Detail & Related papers  (2025-05-27T07:19:37Z)
• Retinal Fundus Multi-Disease Image Classification using Hybrid CNN-Transformer-Ensemble Architectures [0.3277163122167434]
  Our research is motivated by the urgent global issue of a large population affected by retinal diseases.<n>Our primary objective is to develop a comprehensive diagnostic system capable of accurately predicting retinal diseases.
  arXiv  Detail & Related papers  (2025-03-27T12:55:07Z)
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models [49.765466293296186]
  Recent progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new possibilities for interactive diagnostic tools.<n>Med-LVLMs often suffer from factual hallucination, which can lead to incorrect diagnoses.<n>We propose a versatile multimodal RAG system, MMed-RAG, designed to enhance the factuality of Med-LVLMs.
  arXiv  Detail & Related papers  (2024-10-16T23:03:27Z)
• Beyond the Eye: A Relational Model for Early Dementia Detection Using Retinal OCTA Images [43.73298205923969]
  We present a novel PolarNet+ that uses retinal optical coherence tomography angiography ( OCTA) to discriminate early-onset Alzheimer's disease (AD) and mild cognitive impairment (MCI) subjects from controls.<n>Our method first maps OCTA images from Cartesian coordinates to polar coordinates, allowing approximate sub-region calculation.<n>We then introduce a multi-view module to serialize and analyze the images along three dimensions for comprehensive, clinically useful information extraction.
  arXiv  Detail & Related papers  (2024-08-09T15:10:34Z)
• Multi-scale Spatio-temporal Transformer-based Imbalanced Longitudinal Learning for Glaucoma Forecasting from Irregular Time Series Images [45.894671834869975]
  Glaucoma is one of the major eye diseases that leads to progressive optic nerve fiber damage and irreversible blindness. We introduce the Multi-scale Spatio-temporal Transformer Network (MST-former) based on the transformer architecture tailored for sequential image inputs. Our method shows excellent generalization capability on the Alzheimer's Disease Neuroimaging Initiative (ADNI) MRI dataset, with an accuracy of 90.3% for mild cognitive impairment and Alzheimer's disease prediction.
  arXiv  Detail & Related papers  (2024-02-21T02:16:59Z)
• Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading [47.50733518140625]
  Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. We propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading.
  arXiv  Detail & Related papers  (2024-01-17T07:54:49Z)
• An Interpretable Multiple-Instance Approach for the Detection of referable Diabetic Retinopathy from Fundus Images [72.94446225783697]
  We propose a machine learning system for the detection of referable Diabetic Retinopathy in fundus images. By extracting local information from image patches and combining it efficiently through an attention mechanism, our system is able to achieve high classification accuracy. We evaluate our approach on publicly available retinal image datasets, in which it exhibits near state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-03-02T13:14:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.