Learning Through Guidance: Knowledge Distillation for Endoscopic Image Classification

• URL: http://arxiv.org/abs/2308.08731v1
• Date: Thu, 17 Aug 2023 02:02:11 GMT
• Title: Learning Through Guidance: Knowledge Distillation for Endoscopic Image Classification
• Authors: Harshala Gammulle, Yubo Chen, Sridha Sridharan, Travis Klein and Clinton Fookes
• Abstract summary: Endoscopy plays a major role in identifying any underlying abnormalities within the gastrointestinal (GI) tract. Deep learning, specifically Convolution Neural Networks (CNNs) which are designed to perform automatic feature learning without any prior feature engineering, has recently reported great benefits for GI endoscopy image analysis. We investigate three KD-based learning frameworks, response-based, feature-based, and relation-based mechanisms, and introduce a novel multi-head attention-based feature fusion mechanism to support relation-based learning.
• Score: 40.366659911178964
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Endoscopy plays a major role in identifying any underlying abnormalities within the gastrointestinal (GI) tract. There are multiple GI tract diseases that are life-threatening, such as precancerous lesions and other intestinal cancers. In the usual process, a diagnosis is made by a medical expert which can be prone to human errors and the accuracy of the test is also entirely dependent on the expert's level of experience. Deep learning, specifically Convolution Neural Networks (CNNs) which are designed to perform automatic feature learning without any prior feature engineering, has recently reported great benefits for GI endoscopy image analysis. Previous research has developed models that focus only on improving performance, as such, the majority of introduced models contain complex deep network architectures with a large number of parameters that require longer training times. However, there is a lack of focus on developing lightweight models which can run in low-resource environments, which are typically encountered in medical clinics. We investigate three KD-based learning frameworks, response-based, feature-based, and relation-based mechanisms, and introduce a novel multi-head attention-based feature fusion mechanism to support relation-based learning. Compared to the existing relation-based methods that follow simplistic aggregation techniques of multi-teacher response/feature-based knowledge, we adopt the multi-head attention technique to provide flexibility towards localising and transferring important details from each teacher to better guide the student. We perform extensive evaluations on two widely used public datasets, KVASIR-V2 and Hyper-KVASIR, and our experimental results signify the merits of our proposed relation-based framework in achieving an improved lightweight model (only 51.8k trainable parameters) that can run in a resource-limited environment.

Related papers

• LoRKD: Low-Rank Knowledge Decomposition for Medical Foundation Models [59.961172635689664]
  "Knowledge Decomposition" aims to improve the performance on specific medical tasks. We propose a novel framework named Low-Rank Knowledge Decomposition (LoRKD) LoRKD explicitly separates gradients from different tasks by incorporating low-rank expert modules and efficient knowledge separation convolution.
  arXiv  Detail & Related papers  (2024-09-29T03:56:21Z)
• When Medical Imaging Met Self-Attention: A Love Story That Didn't Quite Work Out [8.113092414596679]
  We extend two widely adopted convolutional architectures with different self-attention variants on two different medical datasets. We observe no significant improvement in balanced accuracy over fully convolutional models. We also find that important features, such as dermoscopic structures in skin lesion images, are still not learned by employing self-attention.
  arXiv  Detail & Related papers  (2024-04-18T16:18:41Z)
• Specialty detection in the context of telemedicine in a highly imbalanced multi-class distribution [3.992328888937568]
  The study focuses on handling multiclass and highly imbalanced datasets for Arabic medical questions. The proposed module is deployed in both synchronous and asynchronous medical consultations.
  arXiv  Detail & Related papers  (2024-02-21T06:39:04Z)
• An Evaluation of Lightweight Deep Learning Techniques in Medical Imaging for High Precision COVID-19 Diagnostics [0.0]
  Decision support systems relax the challenges inherent to the physical examination of images. Most deep learning algorithms utilised approaches are not amenable to implementation on resource-constrained devices. This paper presents the development and evaluation of the performance of lightweight deep learning technique for the detection of COVID-19 using the MobileNetV2 model.
  arXiv  Detail & Related papers  (2023-05-30T13:14:03Z)
• Federated Learning with Research Prototypes for Multi-Center MRI-based Detection of Prostate Cancer with Diverse Histopathology [3.8613414331251423]
  We introduce a flexible federated learning framework for cross-site training, validation, and evaluation of deep prostate cancer detection algorithms. Our results show increases in prostate cancer detection and classification accuracy using a specialized neural network model and diverse prostate biopsy data. We open-source our FLtools system, which can be easily adapted to other deep learning projects for medical imaging.
  arXiv  Detail & Related papers  (2022-06-11T21:28:17Z)
• SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images [62.60956024215873]
  Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. This study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification.
  arXiv  Detail & Related papers  (2022-03-22T06:54:29Z)
• Relational Subsets Knowledge Distillation for Long-tailed Retinal Diseases Recognition [65.77962788209103]
  We propose class subset learning by dividing the long-tailed data into multiple class subsets according to prior knowledge. It enforces the model to focus on learning the subset-specific knowledge. The proposed framework proved to be effective for the long-tailed retinal diseases recognition task.
  arXiv  Detail & Related papers  (2021-04-22T13:39:33Z)
• Multi-Disease Detection in Retinal Imaging based on Ensembling Heterogeneous Deep Learning Models [0.0]
  We propose an innovative multi-disease detection pipeline for retinal imaging. Our pipeline includes state-of-the-art strategies like transfer learning, class weighting, real-time image augmentation and Focal loss utilization.
  arXiv  Detail & Related papers  (2021-03-26T18:02:17Z)
• Explaining Clinical Decision Support Systems in Medical Imaging using Cycle-Consistent Activation Maximization [112.2628296775395]
  Clinical decision support using deep neural networks has become a topic of steadily growing interest. clinicians are often hesitant to adopt the technology because its underlying decision-making process is considered to be intransparent and difficult to comprehend. We propose a novel decision explanation scheme based on CycleGAN activation which generates high-quality visualizations of classifier decisions even in smaller data sets.
  arXiv  Detail & Related papers  (2020-10-09T14:39:27Z)
• Select-ProtoNet: Learning to Select for Few-Shot Disease Subtype Prediction [55.94378672172967]
  We focus on few-shot disease subtype prediction problem, identifying subgroups of similar patients. We introduce meta learning techniques to develop a new model, which can extract the common experience or knowledge from interrelated clinical tasks. Our new model is built upon a carefully designed meta-learner, called Prototypical Network, that is a simple yet effective meta learning machine for few-shot image classification.
  arXiv  Detail & Related papers  (2020-09-02T02:50:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.