Categorical Relation-Preserving Contrastive Knowledge Distillation for
Medical Image Classification
- URL: http://arxiv.org/abs/2107.03225v1
- Date: Wed, 7 Jul 2021 13:56:38 GMT
- Title: Categorical Relation-Preserving Contrastive Knowledge Distillation for
Medical Image Classification
- Authors: Xiaohan Xing, Yuenan Hou, Hang Li, Yixuan Yuan, Hongsheng Li, Max
Q.-H. Meng
- Abstract summary: We propose a novel Categorical Relation-preserving Contrastive Knowledge Distillation (CRCKD) algorithm, which takes the commonly used mean-teacher model as the supervisor.
With this regularization, the feature distribution of the student model shows higher intra-class similarity and inter-class variance.
With the contribution of the CCD and CRP, our CRCKD algorithm can distill the relational knowledge more comprehensively.
- Score: 75.27973258196934
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The amount of medical images for training deep classification models is
typically very scarce, making these deep models prone to overfit the training
data. Studies showed that knowledge distillation (KD), especially the
mean-teacher framework which is more robust to perturbations, can help mitigate
the over-fitting effect. However, directly transferring KD from computer vision
to medical image classification yields inferior performance as medical images
suffer from higher intra-class variance and class imbalance. To address these
issues, we propose a novel Categorical Relation-preserving Contrastive
Knowledge Distillation (CRCKD) algorithm, which takes the commonly used
mean-teacher model as the supervisor. Specifically, we propose a novel
Class-guided Contrastive Distillation (CCD) module to pull closer positive
image pairs from the same class in the teacher and student models, while
pushing apart negative image pairs from different classes. With this
regularization, the feature distribution of the student model shows higher
intra-class similarity and inter-class variance. Besides, we propose a
Categorical Relation Preserving (CRP) loss to distill the teacher's relational
knowledge in a robust and class-balanced manner. With the contribution of the
CCD and CRP, our CRCKD algorithm can distill the relational knowledge more
comprehensively. Extensive experiments on the HAM10000 and APTOS datasets
demonstrate the superiority of the proposed CRCKD method.
Related papers
- DCD: Discriminative and Consistent Representation Distillation [6.24302896438145]
We propose Discriminative and Consistent Distillation (DCD) to transfer knowledge from a large teacher model to a smaller student model.
DCD employs a contrastive loss along with a consistency regularization to minimize the discrepancy between the distributions of teacher and student representations.
We show that DCD achieves state-of-the-art performance, with the student model sometimes surpassing the teacher's accuracy.
arXiv Detail & Related papers (2024-07-16T14:53:35Z) - Understanding the Detrimental Class-level Effects of Data Augmentation [63.1733767714073]
achieving optimal average accuracy comes at the cost of significantly hurting individual class accuracy by as much as 20% on ImageNet.
We present a framework for understanding how DA interacts with class-level learning dynamics.
We show that simple class-conditional augmentation strategies improve performance on the negatively affected classes.
arXiv Detail & Related papers (2023-12-07T18:37:43Z) - SINCERE: Supervised Information Noise-Contrastive Estimation REvisited [5.004880836963827]
Previous work suggests a supervised contrastive (SupCon) loss to extend InfoNCE to learn from available class labels.
We propose the Supervised InfoNCE REvisited (SINCERE) loss as a theoretically-justified supervised extension of InfoNCE.
Experiments show that SINCERE leads to better separation of embeddings from different classes and improves transfer learning classification accuracy.
arXiv Detail & Related papers (2023-09-25T16:40:56Z) - Performance of GAN-based augmentation for deep learning COVID-19 image
classification [57.1795052451257]
The biggest challenge in the application of deep learning to the medical domain is the availability of training data.
Data augmentation is a typical methodology used in machine learning when confronted with a limited data set.
In this work, a StyleGAN2-ADA model of Generative Adversarial Networks is trained on the limited COVID-19 chest X-ray image set.
arXiv Detail & Related papers (2023-04-18T15:39:58Z) - Rethinking Semi-Supervised Medical Image Segmentation: A
Variance-Reduction Perspective [51.70661197256033]
We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation.
We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks.
We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
arXiv Detail & Related papers (2023-02-03T13:50:25Z) - SSD-KD: A Self-supervised Diverse Knowledge Distillation Method for
Lightweight Skin Lesion Classification Using Dermoscopic Images [62.60956024215873]
Skin cancer is one of the most common types of malignancy, affecting a large population and causing a heavy economic burden worldwide.
Most studies in skin cancer detection keep pursuing high prediction accuracies without considering the limitation of computing resources on portable devices.
This study specifically proposes a novel method, termed SSD-KD, that unifies diverse knowledge into a generic KD framework for skin diseases classification.
arXiv Detail & Related papers (2022-03-22T06:54:29Z) - Cross-Site Severity Assessment of COVID-19 from CT Images via Domain
Adaptation [64.59521853145368]
Early and accurate severity assessment of Coronavirus disease 2019 (COVID-19) based on computed tomography (CT) images offers a great help to the estimation of intensive care unit event.
To augment the labeled data and improve the generalization ability of the classification model, it is necessary to aggregate data from multiple sites.
This task faces several challenges including class imbalance between mild and severe infections, domain distribution discrepancy between sites, and presence of heterogeneous features.
arXiv Detail & Related papers (2021-09-08T07:56:51Z) - Similarity Transfer for Knowledge Distillation [25.042405967561212]
Knowledge distillation is a popular paradigm for learning portable neural networks by transferring the knowledge from a large model into a smaller one.
We propose a novel method called similarity transfer for knowledge distillation (STKD), which aims to fully utilize the similarities between categories of multiple samples.
It shows that STKD substantially has outperformed the vanilla knowledge distillation and has achieved superior accuracy over the state-of-the-art knowledge distillation methods.
arXiv Detail & Related papers (2021-03-18T06:54:59Z) - Cost-Sensitive Regularization for Diabetic Retinopathy Grading from Eye
Fundus Images [20.480034690570196]
We propose a straightforward approach to enforce the constraint for the task of predicting Diabetic Retinopathy (DR) severity from eye fundus images.
We expand standard classification losses with an extra term that acts as a regularizer.
We show how to adapt our method to the modelling of label noise in each of the sub-problems associated to DR grading.
arXiv Detail & Related papers (2020-10-01T10:42:06Z) - Synergic Adversarial Label Learning for Grading Retinal Diseases via
Knowledge Distillation and Multi-task Learning [29.46896757506273]
Well-qualified doctors annotated images are very expensive and only a limited amount of data is available for various retinal diseases.
Some studies show that AMD and DR share some common features like hemorrhagic points and exudation but most classification algorithms only train those disease models independently.
We propose a method called synergic adversarial label learning (SALL) which leverages relevant retinal disease labels in both semantic and feature space as additional signals and train the model in a collaborative manner.
arXiv Detail & Related papers (2020-03-24T01:32:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.