From Pixels to Explanations: Interpretable Diabetic Retinopathy Grading with CNN-Transformer Ensembles, Visual Explainability and Vision-Language Models

• URL: http://arxiv.org/abs/2604.23079v1
• Date: Sat, 25 Apr 2026 00:21:11 GMT
• Title: From Pixels to Explanations: Interpretable Diabetic Retinopathy Grading with CNN-Transformer Ensembles, Visual Explainability and Vision-Language Models
• Abstract summary: This study presents a methodology that combines strong discriminative models with multimodal explanations.<n>For interpretability, we produced Grad-CAM++ visual attribution maps and short textual rationales.<n>For explanation quality, Grad-CAM++ offered plausible but coarse localization, and VLM rationales were generally grade-consistent.
• Score: 7.782282985072339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The quality of diabetic retinopathy (DR) screening relies on the ability to correctly grade severity; however, many deep-learning (DL) classifiers cannot be easily interpreted in the clinical context. This study presents a methodology that combines strong discriminative models with multimodal explanations, converting retinal pixels into clinically interpretable outputs. Using the APTOS 2019 benchmark, we evaluated six representative CNN- and transformer-based backbones under a controlled protocol with stratified five-fold cross-validation. We then compared ensembling strategies (hard voting, weighted soft voting, stacking) and investigated a hybrid class-level fusion variant to exploit grade-specific advantages. For interpretability, we produced Grad-CAM++ visual attribution maps and short textual rationales using vision-language models (VLMs) conditioned on the fundus image and classifier outputs under conservative prompting constraints. Modern CNN backbones (ResNet-50 and ConvNeXt-Tiny) provided the strongest single-model baselines, with cross-validated QWK up to 0.919 and 0.914, respectively. Ensembling improved ordinal agreement, and weighted soft voting was the most consistent across folds (QWK 0.934 +/- 0.017). Hybrid class-level fusion was competitive but did not yield a statistically reliable improvement over standard fusion in paired fold comparisons (Holm-adjusted p >= 1.000). For explanation quality, Grad-CAM++ offered plausible but coarse localization, and VLM rationales were generally grade-consistent. Quantitatively, VLM variants showed a trade-off between clinical completeness and template-level semantic similarity (coverage 0.700 vs. BERTScore 0.072), while image-text alignment was comparable (CLIPScore approximately 0.34).

Related papers

• ORViT-DR: Ordinally-Robust Hybrid ViT for Low-Resolution Diabetic Retinopathy Grading [0.0]
  ORViT-DR, a hybrid deep learning framework, is designed to improve Diabetic retinopathy grading from low-resolution retinal images.<n>The proposed approach combines convolutional feature extraction with transformer-based global context modeling.<n>The training strategy applies progressive layer unfreezing, layer-wise learning rate decay, exponential moving average (EMA) parameter updates, and ensemble-based prediction.
  arXiv  Detail & Related papers  (2026-08-16T16:35:07Z)
• Vision-Language Models as Zero-Annotation Oracles in Histopathology [10.853300035340615]
  Foreground segmentation is the critical first step of every computational pathology pipeline.<n>We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task.<n>Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.
  arXiv  Detail & Related papers  (2026-06-15T12:50:19Z)
• Geometry-Aware Distillation for Prompt Tuning Biomedical Vision-Language Models [40.432049062995326]
  Current prompt-based and adapter-based tuning of vision-language models (VLMs) is attractive for medical imaging.<n>We propose Omni-Geometry Knowledge Distillation (OGKD), a new framework that injects class-relation structure into the teacher to produce directional targets.<n>Our OGKD achieves substantially better performance, consistently improving accuracy by an average absolute gain of 1.7%-2.8% over all prior state-of-the-art VLM adaptation counterparts.
  arXiv  Detail & Related papers  (2026-06-03T14:17:57Z)
• Benchmarking Convolutional, Transformer, Hybrid, and Vision Language Models for Multi Disease Retinal Screening [0.0]
  We benchmark twelve architectures across four model families: convolutional neural networks, vision transformers, hybrid CNN-transformer backbones, and vision-language models.<n>We evaluate two tasks: binary screening for any retinal disease and multi-label classification across 28 disease classes.<n>SwinTiny and the hybrid CoAtNet0 and MaxViTTiny models achieve the strongest binary screening results and improve macro and micro F1 in the multi-label setting.
  arXiv  Detail & Related papers  (2026-05-25T19:09:35Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• VIVID-Med: LLM-Supervised Structured Pretraining for Deployable Medical ViTs [24.283989257873085]
  We introduce VIVID-Med, a novel framework that leverages a frozen large language model (LLM) as a structured semantic teacher to pretrain medical vision transformers (ViTs)<n> VIVID-Med translates clinical findings into verifiable field-state pairs via a Unified MedicalNIST (UMS), utilizing answerability-aware masking to focus optimization.<n>It achieves a macro-AUC of 0.8588, outperforming BiomedCLIP by +6.65 points while using 500x less data.
  arXiv  Detail & Related papers  (2026-03-10T02:42:51Z)
• AgriPath: A Systematic Exploration of Architectural Trade-offs for Crop Disease Classification [45.52399819498234]
  This work presents a systematic empirical comparison of three model paradigms for fine-grained crop disease classification.<n>We introduce AgriPath-LF16, a benchmark containing 111k images spanning 16 crops and 41 diseases.<n>CNNs achieve the highest accuracy on lab imagery but degrade under domain shift.<n> Contrastive VLMs provide a robust and parameter-efficient alternative with competitive cross-domain performance.
  arXiv  Detail & Related papers  (2026-03-08T17:28:01Z)
• Synthetic Vasculature and Pathology Enhance Vision-Language Model Reasoning [39.96133625333846]
  We introduce Synthetic Vasculature Reasoning (SVR), a framework that controllably synthesizes images and corresponding text.<n>Based on this we curate OCTA-100K-SVR, an OCTA image-reasoning dataset with 100,000 pairs.<n>Our experiments show that a general-purpose VLM trained on the dataset achieves a zero-shot balanced classification accuracy of 89.67% on real OCTA images.
  arXiv  Detail & Related papers  (2025-12-11T19:19:39Z)
• MelanomaNet: Explainable Deep Learning for Skin Lesion Classification [0.0]
  We present MelanomaNet, an explainable deep learning system for skin lesion classification.<n>We evaluate our system on the ISIC 2019 dataset containing 25,331 dermoscopic images.<n>Our results demonstrate that high classification can be achieved alongside comprehensive interpretability.
  arXiv  Detail & Related papers  (2025-12-10T03:22:44Z)
• ConMatFormer: A Multi-attention and Transformer Integrated ConvNext based Deep Learning Model for Enhanced Diabetic Foot Ulcer Classification [0.21990652930491858]
  We propose ConMatFormer, a new hybrid deep learning architecture that combines ConvNeXt blocks, multiple attention mechanisms and transformer modules.<n>Tests showed that ConMatFormer outperformed state-of-the-art (SOTA) convolutional neural network (CNN) and Vision Transformer (ViT) models in terms of accuracy, reliability, and flexibility.<n>Our findings set a new benchmark for DFU classification and provide a hybrid attention transformer framework for medical image analysis.
  arXiv  Detail & Related papers  (2025-10-26T16:34:43Z)
• Glo-VLMs: Leveraging Vision-Language Models for Fine-Grained Diseased Glomerulus Classification [7.87247433522498]
  We introduce Glo-VLMs, a systematic framework designed to explore the adaptation of vision-language models to fine-grained glomerular classification.<n>Our approach leverages curated pathology images alongside clinical text prompts to facilitate joint image-text representation learning.<n>To ensure a fair comparison, we evaluate all models using standardized multi-class metrics, aiming to clarify the practical requirements and potential of large pretrained models for specialized clinical research applications.
  arXiv  Detail & Related papers  (2025-08-21T21:05:44Z)
• Comparison of ConvNeXt and Vision-Language Models for Breast Density Assessment in Screening Mammography [39.58317527488534]
  This study compares multimodal and CNN-based methods for automated classification using the BI-RADS system.<n>Zero-shot classification achieved modest performance, while the fine-tuned ConvNeXt model outperformed the BioMedCLIP linear probe.<n>These findings suggest that despite the promise of multimodal learning, CNN-based models with end-to-end fine-tuning provide stronger performance for specialized medical imaging.
  arXiv  Detail & Related papers  (2025-06-16T20:14:37Z)
• KaLDeX: Kalman Filter based Linear Deformable Cross Attention for Retina Vessel Segmentation [46.57880203321858]
  We propose a novel network (KaLDeX) for vascular segmentation leveraging a Kalman filter based linear deformable cross attention (LDCA) module. Our approach is based on two key components: Kalman filter (KF) based linear deformable convolution (LD) and cross-attention (CA) modules. The proposed method is evaluated on retinal fundus image datasets (DRIVE, CHASE_BD1, and STARE) as well as the 3mm and 6mm of the OCTA-500 dataset.
  arXiv  Detail & Related papers  (2024-10-28T16:00:42Z)
• Vision Transformers for femur fracture classification [59.99241204074268]
  The Vision Transformer (ViT) was able to correctly predict 83% of the test images. Good results were obtained in sub-fractures with the largest and richest dataset ever.
  arXiv  Detail & Related papers  (2021-08-07T10:12:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.