A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation

• URL: http://arxiv.org/abs/2307.12574v1
• Date: Mon, 24 Jul 2023 07:46:06 GMT
• Title: A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation
• Authors: Jinjing Zhu, Yunhao Luo, Xu Zheng, Hao Wang and Lin Wang
• Abstract summary: We propose an online knowledge distillation (KD) framework that can simultaneously learn CNN-based and ViT-based models. Our proposed framework outperforms the state-of-the-art online distillation methods by a large margin.
• Score: 8.110815355364947
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we strive to answer the question "how to collaboratively learn convolutional neural network (CNN)-based and vision transformer (ViT)-based models by selecting and exchanging the reliable knowledge between them for semantic segmentation?" Accordingly, we propose an online knowledge distillation (KD) framework that can simultaneously learn compact yet effective CNN-based and ViT-based models with two key technical breakthroughs to take full advantage of CNNs and ViT while compensating their limitations. Firstly, we propose heterogeneous feature distillation (HFD) to improve students' consistency in low-layer feature space by mimicking heterogeneous features between CNNs and ViT. Secondly, to facilitate the two students to learn reliable knowledge from each other, we propose bidirectional selective distillation (BSD) that can dynamically transfer selective knowledge. This is achieved by 1) region-wise BSD determining the directions of knowledge transferred between the corresponding regions in the feature space and 2) pixel-wise BSD discerning which of the prediction knowledge to be transferred in the logit space. Extensive experiments on three benchmark datasets demonstrate that our proposed framework outperforms the state-of-the-art online distillation methods by a large margin, and shows its efficacy in learning collaboratively between ViT-based and CNN-based models.

Related papers

• CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation [60.08541107831459]
  This paper proposes a CNN-Transformer rectified collaborative learning framework to learn stronger CNN-based and Transformer-based models for medical image segmentation. Specifically, we propose a rectified logit-wise collaborative learning (RLCL) strategy which introduces the ground truth to adaptively select and rectify the wrong regions in student soft labels. We also propose a class-aware feature-wise collaborative learning (CFCL) strategy to achieve effective knowledge transfer between CNN-based and Transformer-based models in the feature space.
  arXiv  Detail & Related papers  (2024-08-25T01:27:35Z)
• Learning CNN on ViT: A Hybrid Model to Explicitly Class-specific Boundaries for Domain Adaptation [13.753795233064695]
  Most domain adaptation (DA) methods are based on either a convolutional neural networks (CNNs) or a vision transformers (ViTs) We design a hybrid method to fully take advantage of both ViT and CNN, called Explicitly Class-specific Boundaries (ECB) ECB learns CNN on ViT to combine their distinct strengths.
  arXiv  Detail & Related papers  (2024-03-27T08:52:44Z)
• Direct Distillation between Different Domains [97.39470334253163]
  We propose a new one-stage method dubbed Direct Distillation between Different Domains" (4Ds) We first design a learnable adapter based on the Fourier transform to separate the domain-invariant knowledge from the domain-specific knowledge. We then build a fusion-activation mechanism to transfer the valuable domain-invariant knowledge to the student network.
  arXiv  Detail & Related papers  (2024-01-12T02:48:51Z)
• Distilling Efficient Vision Transformers from CNNs for Semantic Segmentation [12.177329445930276]
  We propose a novel CNN-to-ViT KD framework, dubbed C2VKD. We first propose a novel visual-linguistic feature distillation (VLFD) module that explores efficient KD among the aligned visual and linguistic-compatible representations. We then propose a pixel-wise decoupled distillation (PDD) module to supervise the student under the combination of labels and teacher's predictions from the decoupled target and non-target classes.
  arXiv  Detail & Related papers  (2023-10-11T07:45:37Z)
• Transformer-CNN Cohort: Semi-supervised Semantic Segmentation by the Best of Both Students [18.860732413631887]
  We propose a novel Semi-supervised Learning (SSL) approach that consists of two students with one based on the vision transformer (ViT) and the other based on the convolutional neural network (CNN) Our method subtly incorporates the multi-level consistency regularization on the predictions and the heterogeneous feature spaces via pseudo-labeling for the unlabeled data. We validate the TCC framework on Cityscapes and Pascal VOC 2012 datasets, which outperforms existing SSL methods by a large margin.
  arXiv  Detail & Related papers  (2022-09-06T02:11:08Z)
• Cross-Architecture Knowledge Distillation [32.689574589575244]
  It is natural to distill complementary knowledge from Transformer to convolutional neural network (CNN) To deal with this problem, a novel cross-architecture knowledge distillation method is proposed. The proposed method outperforms 14 state-of-the-arts on both small-scale and large-scale datasets.
  arXiv  Detail & Related papers  (2022-07-12T02:50:48Z)
• Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillation [91.56643684860062]
  Inter-Channel Correlation for Knowledge Distillation(ICKD) is developed. ICKD captures intrinsic distribution of the featurespace and sufficient diversity properties of features in the teacher network. We are the first method based on knowl-edge distillation boosts ResNet18 beyond 72% Top-1 ac-curacy on ImageNet classification.
  arXiv  Detail & Related papers  (2022-02-08T07:01:56Z)
• Transfer Learning in Multi-Agent Reinforcement Learning with Double Q-Networks for Distributed Resource Sharing in V2X Communication [24.442174952832108]
  This paper addresses the problem of decentralized spectrum sharing in vehicle-to-everything (V2X) communication networks. The aim is to provide resource-efficient coexistence of vehicle-to-infrastructure(V2I) and vehicle-to-vehicle(V2V) links.
  arXiv  Detail & Related papers  (2021-07-13T15:50:10Z)
• Knowledge Distillation By Sparse Representation Matching [107.87219371697063]
  We propose Sparse Representation Matching (SRM) to transfer intermediate knowledge from one Convolutional Network (CNN) to another by utilizing sparse representation. We formulate as a neural processing block, which can be efficiently optimized using gradient descent and integrated into any CNN in a plug-and-play manner. Our experiments demonstrate that is robust to architectural differences between the teacher and student networks, and outperforms other KD techniques across several datasets.
  arXiv  Detail & Related papers  (2021-03-31T11:47:47Z)
• PIN: A Novel Parallel Interactive Network for Spoken Language Understanding [68.53121591998483]
  In the existing RNN-based approaches, ID and SF tasks are often jointly modeled to utilize the correlation information between them. The experiments on two benchmark datasets, i.e., SNIPS and ATIS, demonstrate the effectiveness of our approach. More encouragingly, by using the feature embedding of the utterance generated by the pre-trained language model BERT, our method achieves the state-of-the-art among all comparison approaches.
  arXiv  Detail & Related papers  (2020-09-28T15:59:31Z)
• Adversarial Bipartite Graph Learning for Video Domain Adaptation [50.68420708387015]
  Domain adaptation techniques, which focus on adapting models between distributionally different domains, are rarely explored in the video recognition area. Recent works on visual domain adaptation which leverage adversarial learning to unify the source and target video representations are not highly effective on the videos. This paper proposes an Adversarial Bipartite Graph (ABG) learning framework which directly models the source-target interactions.
  arXiv  Detail & Related papers  (2020-07-31T03:48:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.