Evaluating the Explainability of Vision Transformers in Medical Imaging

• URL: http://arxiv.org/abs/2510.12021v1
• Date: Mon, 13 Oct 2025 23:53:26 GMT
• Title: Evaluating the Explainability of Vision Transformers in Medical Imaging
• Authors: Leili Barekatain, Ben Glocker,
• Abstract summary: This study evaluates the explainability of different Vision Transformer architectures and pre-training strategies.<n>We conduct both quantitative and qualitative analyses on two medical imaging tasks: peripheral blood cell classification and breast ultrasound image classification.<n>Our findings indicate that DINO combined with Grad-CAM offers the most faithful and localized explanations across datasets.
• Score: 10.88831138993597
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding model decisions is crucial in medical imaging, where interpretability directly impacts clinical trust and adoption. Vision Transformers (ViTs) have demonstrated state-of-the-art performance in diagnostic imaging; however, their complex attention mechanisms pose challenges to explainability. This study evaluates the explainability of different Vision Transformer architectures and pre-training strategies - ViT, DeiT, DINO, and Swin Transformer - using Gradient Attention Rollout and Grad-CAM. We conduct both quantitative and qualitative analyses on two medical imaging tasks: peripheral blood cell classification and breast ultrasound image classification. Our findings indicate that DINO combined with Grad-CAM offers the most faithful and localized explanations across datasets. Grad-CAM consistently produces class-discriminative and spatially precise heatmaps, while Gradient Attention Rollout yields more scattered activations. Even in misclassification cases, DINO with Grad-CAM highlights clinically relevant morphological features that appear to have misled the model. By improving model transparency, this research supports the reliable and explainable integration of ViTs into critical medical diagnostic workflows.

Related papers

• A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI [0.06372261626436675]
  This study presents a hybrid dual-stream deep learning framework built on teacher-student knowledge distillation.<n>A student network was implemented as a compact Tiny-ViT structure that inherits the teacher's semantic and morphological knowledge.<n>Two carefully curated Wireless Capsule Endoscopy datasets, encompassing major GI disease classes, were employed to ensure balanced representation.
  arXiv  Detail & Related papers  (2025-12-24T07:51:54Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• DeepGI: Explainable Deep Learning for Gastrointestinal Image Classification [0.0]
  The study confronts common endoscopic challenges such as variable lighting, fluctuating camera angles, and frequent imaging artifacts.<n>The best performing models, VGG16 and MobileNetV2, each achieved a test accuracy of 96.5%.<n>The approach includes explainable AI via Grad-CAM visualization, enabling identification of image regions most influential to model predictions.
  arXiv  Detail & Related papers  (2025-11-26T22:35:57Z)
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• Brain Stroke Detection and Classification Using CT Imaging with Transformer Models and Explainable AI [0.0]
  This study proposed an artificial intelligence framework for multiclass stroke classification using CT scan images.<n>The proposed method adopted MaxViT, a state-of-the-art Vision Transformer, as the primary deep learning model for image-based stroke classification.<n>To enhance model generalization and address class imbalance, we applied data augmentation techniques.
  arXiv  Detail & Related papers  (2025-07-13T13:50:50Z)
• Fine-tuning Vision Language Models with Graph-based Knowledge for Explainable Medical Image Analysis [44.0659716298839]
  Current staging models for Diabetic Retinopathy (DR) are hardly interpretable.<n>We present a novel method that integrates graph representation learning with vision-language models (VLMs) to deliver explainable DR diagnosis.
  arXiv  Detail & Related papers  (2025-03-12T20:19:07Z)
• Multi-Scale Transformer Architecture for Accurate Medical Image Classification [4.578375402082224]
  This study introduces an AI-driven skin lesion classification algorithm built on an enhanced Transformer architecture.<n>By integrating a multi-scale feature fusion mechanism and refining the self-attention process, the model effectively extracts both global and local features.<n>Performance evaluation on the ISIC 2017 dataset demonstrates that the improved Transformer surpasses established AI models.
  arXiv  Detail & Related papers  (2025-02-10T08:22:25Z)
• Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer [4.672688418357066]
  We propose a novel Transformer Diffusion (DTS) model for robust segmentation in the presence of noise. Our model, which analyzes the morphological representation of images, shows better results than the previous models in various medical imaging modalities.
  arXiv  Detail & Related papers  (2024-08-01T07:35:54Z)
• QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge [93.61262892578067]
  Uncertainty in medical image segmentation tasks, especially inter-rater variability, presents a significant challenge. This variability directly impacts the development and evaluation of automated segmentation algorithms. We report the set-up and summarize the benchmark results of the Quantification of Uncertainties in Biomedical Image Quantification Challenge (QUBIQ)
  arXiv  Detail & Related papers  (2024-03-19T17:57:24Z)
• Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework [43.453943987647015]
  Medical vision language pre-training has emerged as a frontier of research, enabling zero-shot pathological recognition. Due to the complex semantics of biomedical texts, current methods struggle to align medical images with key pathological findings in unstructured reports. This is achieved by consulting a large language model and medical experts. Ours improves the accuracy of recent methods by up to 8.56% and 17.26% for seen and unseen categories, respectively.
  arXiv  Detail & Related papers  (2024-03-12T13:18:22Z)
• A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics [63.106382317917344]
  We report a Transformer-based representation-learning model as a clinical diagnostic aid that processes multimodal input in a unified manner. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary diseases.
  arXiv  Detail & Related papers  (2023-06-01T16:23:47Z)
• Towards Evaluating Explanations of Vision Transformers for Medical Imaging [7.812073412066698]
  Vision Transformer (ViT) is a promising alternative to convolutional neural networks for image classification. This paper investigates the performance of various interpretation methods on a ViT applied to classify chest X-ray images.
  arXiv  Detail & Related papers  (2023-04-12T19:37:28Z)
• MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer [53.575573940055335]
  We propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities.
  arXiv  Detail & Related papers  (2023-01-19T03:42:36Z)
• Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [116.87918100031153]
  We propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG) CGT injects clinical relation triples into the visual features as prior knowledge to drive the decoding procedure. Experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods.
  arXiv  Detail & Related papers  (2022-06-04T13:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.