Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentation

• URL: http://arxiv.org/abs/2606.25278v1
• Date: Wed, 24 Jun 2026 01:31:22 GMT
• Title: Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentation
• Abstract summary: Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation.<n>We propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloud-based network through knowledge distillation.<n>The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets.
• Score: 71.94615164565589
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation. Despite the impressive performance, these methods either rely on auxiliary input signals or suffer from costly computational expense. To efficaciously enhance the segmentation performance without introducing intolerable costs, we propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloudbased network through knowledge distillation. Specifically, we present Information-oriented Heterogeneous Distillation (IHD) to help the uni-modal model absorb the complementary knowledge from the multi-modal teacher. We design the Information-Oriented Filtering (IOF) strategy to select informative images from the continuous image sequence for multi-modal fusion. This practice can boost the performance of the multi-modal teacher, thus benefiting the learning of the student. Besides, as opposed to vanilla model ensembling that requires the separate training of each expert, we propose Adept Snapshot Distillation (ASD). ASD treats the freely available model snapshots generated during the training phase as multiple experts, which significantly reduces the training cost for model ensembling. For each expert teacher, it only provides supervision to the student in the class where it is adept. The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets. HAS-KD can be seamlessly integrated into contemporary 3D segmentation algorithms and bring considerable gains without introducing extra inference burdens. The code will be made publicly available upon publication.

Related papers

• Multimodal Robust Prompt Distillation for 3D Point Cloud Models [16.319048523015773]
  Adrial attacks pose a significant threat to learning-based 3D point cloud models.<n>We propose Multimodal Robust Prompt Distillation (MRPD) for distilling robust 3D point cloud model.<n>It learns lightweight prompts by aligning student point cloud model's features with robust embeddings from three distinct teachers.
  arXiv  Detail & Related papers  (2025-11-26T16:49:38Z)
• Learning from Stochastic Teacher Representations Using Student-Guided Knowledge Distillation [64.15918654558816]
  Self-distillation (SSD) training strategy is introduced for filtering and weighting teacher representation to distill from task-relevant representations only.<n> Experimental results on real-world affective computing, wearable/biosignal datasets from the UCR Archive, the HAR dataset, and image classification datasets show that the proposed SSD method can outperform state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-04-19T14:08:56Z)
• Reinforced Multi-teacher Knowledge Distillation for Efficient General Image Forgery Detection and Localization [9.721443347546876]
  Image forgery detection and localization (IFDL) is of vital importance as forged images can spread misinformation that poses potential threats to our daily lives.<n>Previous methods still struggled to effectively handle forged images processed with diverse forgery operations in real-world scenarios.<n>We propose a novel Reinforced Multi-teacher Knowledge Distillation (Re-MTKD) framework for the IFDL task, structured around an encoder-decoder textbfConvNeXt-textbfUperNet.
  arXiv  Detail & Related papers  (2025-04-07T16:12:05Z)
• Is Contrastive Distillation Enough for Learning Comprehensive 3D Representations? [55.99654128127689]
  Cross-modal contrastive distillation has recently been explored for learning effective 3D representations.<n>Existing methods focus primarily on modality-shared features, neglecting the modality-specific features during the pre-training process.<n>We propose a new framework, namely CMCR, to address these shortcomings.
  arXiv  Detail & Related papers  (2024-12-12T06:09:49Z)
• Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection [66.74183705987276]
  We introduce a framework to improve the camera-only apprentice model, including an apprentice-friendly multi-modal expert and temporal-fusion-friendly distillation supervision. With those improvements, our camera-only apprentice VCD-A sets new state-of-the-art on nuScenes with a score of 63.1% NDS.
  arXiv  Detail & Related papers  (2023-10-24T09:29:26Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)
• SimDistill: Simulated Multi-modal Distillation for BEV 3D Object Detection [56.24700754048067]
  Multi-view camera-based 3D object detection has become popular due to its low cost, but accurately inferring 3D geometry solely from camera data remains challenging. We propose a Simulated multi-modal Distillation (SimDistill) method by carefully crafting the model architecture and distillation strategy. Our SimDistill can learn better feature representations for 3D object detection while maintaining a cost-effective camera-only deployment.
  arXiv  Detail & Related papers  (2023-03-29T16:08:59Z)
• Online Knowledge Distillation via Multi-branch Diversity Enhancement [15.523646047674717]
  We propose a new distillation method to enhance the diversity among multiple student models. We use Feature Fusion Module (FFM), which improves the performance of the attention mechanism in the network. We also use Diversification(CD) loss function to strengthen the differences between the student models.
  arXiv  Detail & Related papers  (2020-10-02T05:52:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.