Towards Complementary Knowledge Distillation for Efficient Dense Image Prediction

• URL: http://arxiv.org/abs/2401.13174v2
• Date: Mon, 02 Dec 2024 02:55:29 GMT
• Title: Towards Complementary Knowledge Distillation for Efficient Dense Image Prediction
• Authors: Dong Zhang, Pingcheng Dong, Xinting Hu, Long Chen, Kwang-Ting Cheng,
• Abstract summary: It has been revealed that small efficient dense image prediction (EDIP) models, trained using the knowledge distillation (KD) framework, encounter two key challenges. We propose a complementary boundary and context distillation (BCD) method within the KD framework for EDIPs. Our method can outperform existing methods without requiring extra supervisions or incurring increased inference costs.
• Score: 30.480160608154367
• License:
• Abstract: It has been revealed that small efficient dense image prediction (EDIP) models, trained using the knowledge distillation (KD) framework, encounter two key challenges, including maintaining boundary region completeness and preserving target region connectivity, despite their favorable capacity to recognize main object regions. In this work, we propose a complementary boundary and context distillation (BCD) method within the KD framework for EDIPs, which facilitates the targeted knowledge transfer from large accurate teacher models to compact efficient student models. Specifically, the boundary distillation component focuses on extracting explicit object-level semantic boundaries from the hierarchical feature maps of the backbone network to enhance the student model's mask quality in boundary regions. Concurrently, the context distillation component leverages self-relations as a bridge to transfer implicit pixel-level contexts from the teacher model to the student model, ensuring strong connectivity in target regions. Our proposed BCD method is specifically designed for EDIP tasks and is characterized by its simplicity and efficiency. Extensive experimental results across semantic segmentation, object detection, and instance segmentation on various representative datasets demonstrate that our method can outperform existing methods without requiring extra supervisions or incurring increased inference costs, resulting in well-defined object boundaries and smooth connecting regions.

Related papers

• Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching [53.05954114863596]
  We propose a brand-new Deep Boosting Learning (DBL) algorithm for image-text matching. An anchor branch is first trained to provide insights into the data properties. A target branch is concurrently tasked with more adaptive margin constraints to further enlarge the relative distance between matched and unmatched samples.
  arXiv  Detail & Related papers  (2024-04-28T08:44:28Z)
• Attention-guided Feature Distillation for Semantic Segmentation [8.344263189293578]
  This paper showcases the efficacy of a simple yet powerful method for utilizing refined feature maps to transfer attention. The proposed method has proven to be effective in distilling rich information, outperforming existing methods in semantic segmentation as a dense prediction task.
  arXiv  Detail & Related papers  (2024-03-08T16:57:47Z)
• Background Activation Suppression for Weakly Supervised Object Localization and Semantic Segmentation [84.62067728093358]
  Weakly supervised object localization and semantic segmentation aim to localize objects using only image-level labels. New paradigm has emerged by generating a foreground prediction map to achieve pixel-level localization. This paper presents two astonishing experimental observations on the object localization learning process.
  arXiv  Detail & Related papers  (2023-09-22T15:44:10Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation [20.450568708073767]
  This paper proposes boundary-privileged knowledge distillation (BPKD) for semantic segmentation. BPKD distills the knowledge of the teacher model's body and edges separately to the compact student model. Our experiments demonstrate that the proposed BPKD method provides extensive refinements and aggregation for edge and body regions.
  arXiv  Detail & Related papers  (2023-06-13T18:45:32Z)
• Impact of a DCT-driven Loss in Attention-based Knowledge-Distillation for Scene Recognition [64.29650787243443]
  We propose and analyse the use of a 2D frequency transform of the activation maps before transferring them. This strategy enhances knowledge transferability in tasks such as scene recognition. We publicly release the training and evaluation framework used along this paper at http://www.vpu.eps.uam.es/publications/DCTBasedKDForSceneRecognition.
  arXiv  Detail & Related papers  (2022-05-04T11:05:18Z)
• Point-Level Region Contrast for Object Detection Pre-Training [147.47349344401806]
  We present point-level region contrast, a self-supervised pre-training approach for the task of object detection. Our approach performs contrastive learning by directly sampling individual point pairs from different regions. Compared to an aggregated representation per region, our approach is more robust to the change in input region quality.
  arXiv  Detail & Related papers  (2022-02-09T18:56:41Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)
• Boundary Guided Context Aggregation for Semantic Segmentation [23.709865471981313]
  We exploit boundary as a significant guidance for context aggregation to promote the overall semantic understanding of an image. We conduct extensive experiments on the Cityscapes and ADE20K databases, and comparable results are achieved with the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-10-27T17:04:38Z)
• BiconNet: An Edge-preserved Connectivity-based Approach for Salient Object Detection [3.3517146652431378]
  We show that our model can use any existing saliency-based SOD framework as its backbone. Through comprehensive experiments on five benchmark datasets, we demonstrate that our proposed method outperforms state-of-the-art SOD approaches.
  arXiv  Detail & Related papers  (2021-02-27T21:39:04Z)
• Interpretable and Accurate Fine-grained Recognition via Region Grouping [14.28113520947247]
  We present an interpretable deep model for fine-grained visual recognition. At the core of our method lies the integration of region-based part discovery and attribution within a deep neural network. Our results compare favorably to state-of-the-art methods on classification tasks.
  arXiv  Detail & Related papers  (2020-05-21T01:18:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.