Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification

• URL: http://arxiv.org/abs/2107.03225v1
• Date: Wed, 7 Jul 2021 13:56:38 GMT
• Title: Categorical Relation-Preserving Contrastive Knowledge Distillation for Medical Image Classification
• Authors: Xiaohan Xing, Yuenan Hou, Hang Li, Yixuan Yuan, Hongsheng Li, Max Q.-H. Meng
• Abstract summary: We propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, which takes the commonly used mean-teacher model as the supervisor. With this regularization, the feature distribution of the student model shows higher intra-class similarity and inter-class variance. With the contribution of the CCD and CRP, our CRCKD algorithm can distill the relational knowledge more comprehensively.
• Score: 75.27973258196934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The amount of medical images for training deep classification models is typically very scarce, making these deep models prone to overfit the training data. Studies showed that knowledge distillation (KD), especially the mean-teacher framework which is more robust to perturbations, can help mitigate the over-fitting effect. However, directly transferring KD from computer vision to medical image classification yields inferior performance as medical images suffer from higher intra-class variance and class imbalance. To address these issues, we propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, which takes the commonly used mean-teacher model as the supervisor. Specifically, we propose a novel Class-guided Contrastive Distillation (CCD) module to pull closer positive image pairs from the same class in the teacher and student models, while pushing apart negative image pairs from different classes. With this regularization, the feature distribution of the student model shows higher intra-class similarity and inter-class variance. Besides, we propose a Categorical Relation Preserving (CRP) loss to distill the teacher's relational knowledge in a robust and class-balanced manner. With the contribution of the CCD and CRP, our CRCKD algorithm can distill the relational knowledge more comprehensively. Extensive experiments on the HAM10000 and APTOS datasets demonstrate the superiority of the proposed CRCKD method.

Related papers

• DCD: Discriminative and Consistent Representation Distillation [6.24302896438145]
  We propose Discriminative and Consistent Distillation (DCD) to transfer knowledge from a large teacher model to a smaller student model. DCD employs a contrastive loss along with a consistency regularization to minimize the discrepancy between the distributions of teacher and student representations. We show that DCD achieves state-of-the-art performance, with the student model sometimes surpassing the teacher's accuracy.
  arXiv  Detail & Related papers  (2024-07-16T14:53:35Z)
• Understanding the Detrimental Class-level Effects of Data Augmentation [63.1733767714073]
  achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much as 20% on ImageNet. We present a framework for understanding how DA interacts with class-level learning dynamics. We show that simple class-conditional augmentation strategies improve performance on the negatively affected classes.
  arXiv  Detail & Related papers  (2023-12-07T18:37:43Z)
• SINCERE: Supervised Information Noise-Contrastive Estimation REvisited [5.004880836963827]
  Previous work suggests a supervised contrastive (SupCon) loss to extend InfoNCE to learn from available class labels. We propose the Supervised InfoNCE REvisited (SINCERE) loss as a theoretically-justified supervised extension of InfoNCE. Experiments show that SINCERE leads to better separation of embeddings from different classes and improves transfer learning classification accuracy.
  arXiv  Detail & Related papers  (2023-09-25T16:40:56Z)
• Performance of GAN-based augmentation for deep learning COVID-19 image classification [57.1795052451257]
  The biggest challenge in the application of deep learning to the medical domain is the availability of training data. Data augmentation is a typical methodology used in machine learning when confronted with a limited data set. In this work, a StyleGAN2-ADA model of Generative Adversarial Networks is trained on the limited COVID-19 chest X-ray image set.
  arXiv  Detail & Related papers  (2023-04-18T15:39:58Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for Lightweight Skin Lesion Classification Using Dermoscopic Images [62.60956024215873]
  Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide. Most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices. This study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification.
  arXiv  Detail & Related papers  (2022-03-22T06:54:29Z)
• Cross-Site Severity Assessment of COVID-19 from CT Images via Domain Adaptation [64.59521853145368]
  Early and accurate severity assessment of Coronavirus disease 2019 (COVID-19) based on computed tomography (CT) images offers a great help to the estimation of intensive care unit event. To augment the labeled data and improve the generalization ability of the classification model, it is necessary to aggregate data from multiple sites. This task faces several challenges including class imbalance between mild and severe infections, domain distribution discrepancy between sites, and presence of heterogeneous features.
  arXiv  Detail & Related papers  (2021-09-08T07:56:51Z)
• Similarity Transfer for Knowledge Distillation [25.042405967561212]
  Knowledge distillation is a popular paradigm for learning portable neural networks by transferring the knowledge from a large model into a smaller one. We propose a novel method called similarity transfer for knowledge distillation (STKD), which aims to fully utilize the similarities between categories of multiple samples. It shows that STKD substantially has outperformed the vanilla knowledge distillation and has achieved superior accuracy over the state-of-the-art knowledge distillation methods.
  arXiv  Detail & Related papers  (2021-03-18T06:54:59Z)
• Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye Fundus Images [20.480034690570196]
  We propose a straightforward approach to enforce the constraint for the task of predicting Diabetic Retinopathy (DR) severity from eye fundus images. We expand standard classification losses with an extra term that acts as a regularizer. We show how to adapt our method to the modelling of label noise in each of the sub-problems associated to DR grading.
  arXiv  Detail & Related papers  (2020-10-01T10:42:06Z)
• Synergic Adversarial Label Learning for Grading Retinal Diseases via Knowledge Distillation and Multi-task Learning [29.46896757506273]
  Well-qualified doctors annotated images are very expensive and only a limited amount of data is available for various retinal diseases. Some studies show that AMD and DR share some common features like hemorrhagic points and exudation but most classification algorithms only train those disease models independently. We propose a method called synergic adversarial label learning (SALL) which leverages relevant retinal disease labels in both semantic and feature space as additional signals and train the model in a collaborative manner.
  arXiv  Detail & Related papers  (2020-03-24T01:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.