Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images

• URL: http://arxiv.org/abs/2310.01912v1
• Date: Tue, 3 Oct 2023 09:35:38 GMT
• Title: Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images
• Authors: Mostafa El Habib Daho, Yihao Li, Rachid Zeghlache, Yapo Cedric Atse, Hugo Le Boit\'e, Sophie Bonnin, Deborah Cosette, Pierre Deman, Laurent Borderie, Capucine Lepicard, Ramin Tadayoni, B\'eatrice Cochener, Pierre-Henri Conze, Mathieu Lamard, and Gwenol\'e Quellec
• Abstract summary: Diabetic Retinopathy (DR), a prevalent and severe complication of diabetes, affects millions of individuals globally. Recent advancements in imaging technologies provide opportunities for the early detection of DR but also pose significant challenges. This study introduces a novel multimodal approach that leverages these imaging modalities to notably enhance DR classification.
• Score: 1.6449510885987357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diabetic Retinopathy (DR), a prevalent and severe complication of diabetes, affects millions of individuals globally, underscoring the need for accurate and timely diagnosis. Recent advancements in imaging technologies, such as Ultra-WideField Color Fundus Photography (UWF-CFP) imaging and Optical Coherence Tomography Angiography (OCTA), provide opportunities for the early detection of DR but also pose significant challenges given the disparate nature of the data they produce. This study introduces a novel multimodal approach that leverages these imaging modalities to notably enhance DR classification. Our approach integrates 2D UWF-CFP images and 3D high-resolution 6x6 mm$^3$ OCTA (both structure and flow) images using a fusion of ResNet50 and 3D-ResNet50 models, with Squeeze-and-Excitation (SE) blocks to amplify relevant features. Additionally, to increase the model's generalization capabilities, a multimodal extension of Manifold Mixup, applied to concatenated multimodal features, is implemented. Experimental results demonstrate a remarkable enhancement in DR classification performance with the proposed multimodal approach compared to methods relying on a single modality only. The methodology laid out in this work holds substantial promise for facilitating more accurate, early detection of DR, potentially improving clinical outcomes for patients.

Related papers

• Cross-Fundus Transformer for Multi-modal Diabetic Retinopathy Grading with Cataract [17.77175890577782]
  Diabetic retinopathy (DR) is a leading cause of blindness worldwide and a common complication of diabetes. This study explores a novel multi-modal deep learning framework to fuse the information from color fundus photography (IFP) and infrared fundus photography (IFP) towards more accurate DR grading.
  arXiv  Detail & Related papers  (2024-11-01T16:38:49Z)
• BrainMVP: Multi-modal Vision Pre-training for Brain Image Analysis using Multi-parametric MRI [11.569448567735435]
  BrainMVP is a multi-modal vision pre-training framework for brain image analysis using multi-parametric MRI scans. Cross-modal reconstruction is explored to learn distinctive brain image embeddings and efficient modality fusion capabilities. Experiments on downstream tasks demonstrate superior performance compared to state-of-the-art pre-training methods in the medical domain.
  arXiv  Detail & Related papers  (2024-10-14T15:12:16Z)
• Simultaneous Tri-Modal Medical Image Fusion and Super-Resolution using Conditional Diffusion Model [2.507050016527729]
  Tri-modal medical image fusion can provide a more comprehensive view of the disease's shape, location, and biological activity. Due to the limitations of imaging equipment and considerations for patient safety, the quality of medical images is usually limited. There is an urgent need for a technology that can both enhance image resolution and integrate multi-modal information.
  arXiv  Detail & Related papers  (2024-04-26T12:13:41Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• Three-Dimensional Medical Image Fusion with Deformable Cross-Attention [10.26573411162757]
  Multimodal medical image fusion plays an instrumental role in several areas of medical image processing. Traditional fusion methods tend to process each modality independently before combining the features and reconstructing the fusion image. In this study, we introduce an innovative unsupervised feature mutual learning fusion network designed to rectify these limitations.
  arXiv  Detail & Related papers  (2023-10-10T04:10:56Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Deep-OCTA: Ensemble Deep Learning Approaches for Diabetic Retinopathy Analysis on OCTA Images [10.16138081361263]
  We present novel and practical deep-learning solutions based on ultra-wide OCTA for the Diabetic Retinopathy Analysis Challenge (DRAC) In the segmentation of DR lesions task, we utilize UNet and UNet++ to segment three lesions with strong data augmentation and model ensemble. In the image quality assessment task, we create an ensemble of InceptionV3, SE-ResNeXt, and Vision Transformer models.
  arXiv  Detail & Related papers  (2022-10-02T13:23:56Z)
• Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
  We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality. In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network. Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
  arXiv  Detail & Related papers  (2021-10-15T13:16:59Z)
• Multifold Acceleration of Diffusion MRI via Slice-Interleaved Diffusion Encoding (SIDE) [50.65891535040752]
  We propose a diffusion encoding scheme, called Slice-Interleaved Diffusion. SIDE, that interleaves each diffusion-weighted (DW) image volume with slices encoded with different diffusion gradients. We also present a method based on deep learning for effective reconstruction of DW images from the highly slice-undersampled data.
  arXiv  Detail & Related papers  (2020-02-25T14:48:17Z)
• Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis [143.55901940771568]
  We propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis. In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality. A multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality.
  arXiv  Detail & Related papers  (2020-02-11T08:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.