Explainable Knowledge Distillation for Efficient Medical Image Classification

• URL: http://arxiv.org/abs/2508.15251v1
• Date: Thu, 21 Aug 2025 05:22:47 GMT
• Title: Explainable Knowledge Distillation for Efficient Medical Image Classification
• Authors: Aqib Nazir Mir, Danish Raza Rizvi,
• Abstract summary: This study comprehensively explores knowledge distillation frameworks for COVID-19 and lung cancer classification using chest X-ray (CXR) images.<n>We employ high-capacity teacher models, including VGG19 and lightweight Vision Transformers, to guide the training of a compact, hardware-aware student model derived from the OFA-595 supernet.<n>We validate our models on two benchmark datasets: COVID-QU-Ex and LCS25000, covering multiple classes, including COVID-19, healthy, non-COVID pneumonia, lung, and colon cancer.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study comprehensively explores knowledge distillation frameworks for COVID-19 and lung cancer classification using chest X-ray (CXR) images. We employ high-capacity teacher models, including VGG19 and lightweight Vision Transformers (Visformer-S and AutoFormer-V2-T), to guide the training of a compact, hardware-aware student model derived from the OFA-595 supernet. Our approach leverages hybrid supervision, combining ground-truth labels with teacher models' soft targets to balance accuracy and computational efficiency. We validate our models on two benchmark datasets: COVID-QU-Ex and LCS25000, covering multiple classes, including COVID-19, healthy, non-COVID pneumonia, lung, and colon cancer. To interpret the spatial focus of the models, we employ Score-CAM-based visualizations, which provide insight into the reasoning process of both teacher and student networks. The results demonstrate that the distilled student model maintains high classification performance with significantly reduced parameters and inference time, making it an optimal choice in resource-constrained clinical environments. Our work underscores the importance of combining model efficiency with explainability for practical, trustworthy medical AI solutions.

Related papers

• A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI [0.06372261626436675]
  This study presents a hybrid dual-stream deep learning framework built on teacher-student knowledge distillation.<n>A student network was implemented as a compact Tiny-ViT structure that inherits the teacher's semantic and morphological knowledge.<n>Two carefully curated Wireless Capsule Endoscopy datasets, encompassing major GI disease classes, were employed to ensure balanced representation.
  arXiv  Detail & Related papers  (2025-12-24T07:51:54Z)
• KD-OCT: Efficient Knowledge Distillation for Clinical-Grade Retinal OCT Classification [0.0]
  Age-related macular degeneration (AMD) and choroidal neovascularization (CNV)-related conditions are leading causes of vision loss worldwide.<n> deploying state-of-the-art deep learning models like ConvNeXtV2-Large in clinical settings is hindered by their computational demands.<n>It is desirable to develop efficient models that maintain high diagnostic performance while enabling real-time deployment.
  arXiv  Detail & Related papers  (2025-12-09T19:34:30Z)
• A Hybrid Deep Learning CNN Model for Enhanced COVID-19 Detection from Computed Tomography (CT) Scan Images [0.0]
  Early detection of COVID-19 is crucial for effective treatment and controlling its spread.<n>This study proposes a novel hybrid deep learning model for detecting COVID-19 from CT scan images.<n>Our proposed model achieved an accuracy of 98.93%, outperforming the individual models in terms of precision, recall, F1 scores, and ROC curve performance.
  arXiv  Detail & Related papers  (2025-01-28T18:59:21Z)
• Active Data Curation Effectively Distills Large-Scale Multimodal Models [66.23057263509027]
  Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones.<n>In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining.<n>Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations.
  arXiv  Detail & Related papers  (2024-11-27T18:50:15Z)
• EFCM: Efficient Fine-tuning on Compressed Models for deployment of large models in medical image analysis [17.876140405367764]
  This study presents an Efficient Fine-tuning on Compressed Models (EFCM) framework with two stages: unsupervised feature distillation and fine-tuning. Experiments are conducted on 11 downstream datasets related to three large medical models: RETFound for retina, MRM for chest X-ray, and BROW for histopathology.
  arXiv  Detail & Related papers  (2024-09-18T09:08:16Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)
• Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models [4.711401719735324]
  Knowledge Distillation (KD) has been proposed as a compression method and an acceleration technology. KD is an efficient learning strategy that can transfer knowledge from a burdensome model to a lightweight model. We develop a KD-based deep model for prostate MRI segmentation in this work by combining features-based distillation with Kullback-Leibler divergence, Lovasz, and Dice losses.
  arXiv  Detail & Related papers  (2023-03-16T17:15:08Z)
• SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images [62.60956024215873]
  Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. This study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification.
  arXiv  Detail & Related papers  (2022-03-22T06:54:29Z)
• Performance or Trust? Why Not Both. Deep AUC Maximization with Self-Supervised Learning for COVID-19 Chest X-ray Classifications [72.52228843498193]
  In training deep learning models, a compromise often must be made between performance and trust. In this work, we integrate a new surrogate loss with self-supervised learning for computer-aided screening of COVID-19 patients.
  arXiv  Detail & Related papers  (2021-12-14T21:16:52Z)
• Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification [75.27973258196934]
  We propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, which takes the commonly used mean-teacher model as the supervisor. With this regularization, the feature distribution of the student model shows higher intra-class similarity and inter-class variance. With the contribution of the CCD and CRP, our CRCKD algorithm can distill the relational knowledge more comprehensively.
  arXiv  Detail & Related papers  (2021-07-07T13:56:38Z)
• Classification of COVID-19 X-ray Images Using a Combination of Deep and Handcrafted Features [0.0]
  We use a combination of deep convolutional and handcrafted features extracted from X-ray chest scans to discriminate between healthy, common pneumonia, and COVID-19 patients. We achieve an accuracy of 0.988 in the classification task with our combined approach compared to 0.963 and 0.983 accuracy for the handcrafted features with SVM and CNN respectively.
  arXiv  Detail & Related papers  (2021-01-19T21:09:46Z)
• Exploration of Interpretability Techniques for Deep COVID-19 Classification using Chest X-ray Images [10.01138352319106]
  Five different deep learning models (ResNet18, ResNet34, InceptionV3, InceptionResNetV2, and DenseNet161) and their Ensemble have been used in this paper to classify COVID-19, pneumoniae and healthy subjects using Chest X-Ray images. The mean Micro-F1 score of the models for COVID-19 classifications ranges from 0.66 to 0.875, and is 0.89 for the Ensemble of the network models.
  arXiv  Detail & Related papers  (2020-06-03T22:55:53Z)
• Self-Training with Improved Regularization for Sample-Efficient Chest X-Ray Classification [80.00316465793702]
  We present a deep learning framework that enables robust modeling in challenging scenarios. Our results show that using 85% lesser labeled data, we can build predictive models that match the performance of classifiers trained in a large-scale data setting.
  arXiv  Detail & Related papers  (2020-05-03T02:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.