KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp Segmentation

• URL: http://arxiv.org/abs/2312.08555v3
• Date: Tue, 23 Apr 2024 19:31:25 GMT
• Title: KDAS: Knowledge Distillation via Attention Supervision Framework for Polyp Segmentation
• Authors: Quoc-Huy Trinh, Minh-Van Nguyen, Phuoc-Thao Vo Thi,
• Abstract summary: We present KDAS, a Knowledge Distillation framework that incorporates attention supervision, and our proposed Symmetrical Guiding Module. This framework is designed to facilitate a compact student model with fewer parameters, allowing it to learn the strengths of the teacher model. Our compact models demonstrate their strength by achieving competitive results with state-of-the-art methods.
• Score: 6.148777307966648
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Polyp segmentation, a contentious issue in medical imaging, has seen numerous proposed methods aimed at improving the quality of segmented masks. While current state-of-the-art techniques yield impressive results, the size and computational cost of these models create challenges for practical industry applications. To address this challenge, we present KDAS, a Knowledge Distillation framework that incorporates attention supervision, and our proposed Symmetrical Guiding Module. This framework is designed to facilitate a compact student model with fewer parameters, allowing it to learn the strengths of the teacher model and mitigate the inconsistency between teacher features and student features, a common challenge in Knowledge Distillation, via the Symmetrical Guiding Module. Through extensive experiments, our compact models demonstrate their strength by achieving competitive results with state-of-the-art methods, offering a promising approach to creating compact models with high accuracy for polyp segmentation and in the medical imaging field. The implementation is available on https://github.com/huyquoctrinh/KDAS.

Related papers

• RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models [60.596005921295806]
  Agglomerative models have emerged as a powerful approach to training vision foundation models. We identify critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. We propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions.
  arXiv  Detail & Related papers  (2024-12-10T17:06:41Z)
• Active Data Curation Effectively Distills Large-Scale Multimodal Models [66.23057263509027]
  Knowledge distillation (KD) is the de facto standard for compressing large-scale models into smaller ones. In this work we explore an alternative, yet simple approach -- active data curation as effective distillation for contrastive multimodal pretraining. Our simple online batch selection method, ACID, outperforms strong KD baselines across various model-, data- and compute-configurations.
  arXiv  Detail & Related papers  (2024-11-27T18:50:15Z)
• MLLA-UNet: Mamba-like Linear Attention in an Efficient U-Shape Model for Medical Image Segmentation [6.578088710294546]
  Traditional segmentation methods struggle to address challenges such as high anatomical variability, blurred tissue boundaries, low organ contrast, and noise. We propose MLLA-UNet (Mamba-Like Linear Attention UNet), a novel architecture that achieves linear computational complexity while maintaining high segmentation accuracy. Experiments demonstrate that MLLA-UNet achieves state-of-the-art performance on six challenging datasets with 24 different segmentation tasks, including but not limited to FLARE22, AMOS CT, and ACDC, with an average DSC of 88.32%.
  arXiv  Detail & Related papers  (2024-10-31T08:54:23Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• SAM-EG: Segment Anything Model with Egde Guidance framework for efficient Polyp Segmentation [6.709243857842895]
  We propose a framework that guides small segmentation models for polyp segmentation to address the cost challenge. In this study, we introduce the Edge Guiding module, which integrates edge information into image features. Our small models showcase their efficacy by achieving competitive results with state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-06-21T01:42:20Z)
• ECC-PolypDet: Enhanced CenterNet with Contrastive Learning for Automatic Polyp Detection [88.4359020192429]
  Existing methods either involve computationally expensive context aggregation or lack prior modeling of polyps, resulting in poor performance in challenging cases. In this paper, we propose the Enhanced CenterNet with Contrastive Learning (ECC-PolypDet), a two-stage training & end-to-end inference framework. Box-assisted Contrastive Learning (BCL) during training to minimize the intra-class difference and maximize the inter-class difference between foreground polyps and backgrounds, enabling our model to capture concealed polyps. In the fine-tuning stage, we introduce the IoU-guided Sample Re-weighting
  arXiv  Detail & Related papers  (2024-01-10T07:03:41Z)
• Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning [26.425784890859738]
  MaCo is a masked contrastive chest X-ray foundation model. It simultaneously achieves fine-grained image understanding and zero-shot learning for a variety of medical imaging tasks. It is shown to be superior over 10 state-of-the-art approaches across tasks such as classification, segmentation, detection, and phrase grounding.
  arXiv  Detail & Related papers  (2023-09-12T01:29:37Z)
• Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models [4.711401719735324]
  Knowledge Distillation (KD) has been proposed as a compression method and an acceleration technology. KD is an efficient learning strategy that can transfer knowledge from a burdensome model to a lightweight model. We develop a KD-based deep model for prostate MRI segmentation in this work by combining features-based distillation with Kullback-Leibler divergence, Lovasz, and Dice losses.
  arXiv  Detail & Related papers  (2023-03-16T17:15:08Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)
• Demystifying Deep Learning Models for Retinal OCT Disease Classification using Explainable AI [0.6117371161379209]
  The adoption of various deep learning techniques is quite common as well as effective, and its statement is equally true when it comes to implementing it into the retina Optical Coherence Tomography sector. These techniques have the black box characteristics that prevent the medical professionals to completely trust the results generated from them. This paper proposes a self-developed CNN model which is comparatively smaller and simpler along with the use of Lime that introduces Explainable AI to the study.
  arXiv  Detail & Related papers  (2021-11-06T13:54:07Z)
• Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding [60.89561661441736]
  We propose a novel method for few-shot medical image segmentation. We construct our few-shot image segmentor using a deep convolutional network trained episodically. We enhance discriminability of deep embedding to encourage clustering of the feature domains of the same class.
  arXiv  Detail & Related papers  (2020-12-10T04:01:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.