Representation Disparity-aware Distillation for 3D Object Detection

• URL: http://arxiv.org/abs/2308.10308v1
• Date: Sun, 20 Aug 2023 16:06:42 GMT
• Title: Representation Disparity-aware Distillation for 3D Object Detection
• Authors: Yanjing Li, Sheng Xu, Mingbao Lin, Jihao Yin, Baochang Zhang, Xianbin Cao
• Abstract summary: This paper presents a novel representation disparity-aware distillation (RDD) method to address the representation disparity issue. Our RDD increases mAP of CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance while taking up only 42% FLOPs.
• Score: 44.17712259352281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we focus on developing knowledge distillation (KD) for compact 3D detectors. We observe that off-the-shelf KD methods manifest their efficacy only when the teacher model and student counterpart share similar intermediate feature representations. This might explain why they are less effective in building extreme-compact 3D detectors where significant representation disparity arises due primarily to the intrinsic sparsity and irregularity in 3D point clouds. This paper presents a novel representation disparity-aware distillation (RDD) method to address the representation disparity issue and reduce performance gap between compact students and over-parameterized teachers. This is accomplished by building our RDD from an innovative perspective of information bottleneck (IB), which can effectively minimize the disparity of proposal region pairs from student and teacher in features and logits. Extensive experiments are performed to demonstrate the superiority of our RDD over existing KD methods. For example, our RDD increases mAP of CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance while taking up only 42% FLOPs.

Related papers

• Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling [81.00825302340984]
  We introduce Speculative Knowledge Distillation (SKD) to generate high-quality training data on-the-fly. In SKD, the student proposes tokens, and the teacher replaces poorly ranked ones based on its own distribution. We evaluate SKD on various text generation tasks, including translation, summarization, math, and instruction following.
  arXiv  Detail & Related papers  (2024-10-15T06:51:25Z)
• Image-to-Lidar Relational Distillation for Autonomous Driving Data [4.893568782260855]
  2D foundation models excel at addressing 2D tasks with little or no downstream supervision, owing to their robust representations. The emergence of 2D-to-3D distillation frameworks has extended these capabilities to 3D models. But distilling 3D representations for autonomous driving datasets presents challenges like self-similarity, class imbalance, and point cloud sparsity. We propose a relational distillation framework enforcing intra-modal and cross-modal constraints, resulting in distilled 3D representations that closely capture the structure of the 2D representation.
  arXiv  Detail & Related papers  (2024-09-01T21:26:32Z)
• Relative Difficulty Distillation for Semantic Segmentation [54.76143187709987]
  We propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD) RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals. Our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.
  arXiv  Detail & Related papers  (2024-07-04T08:08:25Z)
• Text-to-3D with Classifier Score Distillation [80.14832887529259]
  Classifier-free guidance is considered an auxiliary trick rather than the most essential. We name this method Score Distillation (CSD), which can be interpreted as using an implicit classification model for generation. We validate the effectiveness of CSD across a variety of text-to-3D tasks including shape generation, texture synthesis, and shape editing.
  arXiv  Detail & Related papers  (2023-10-30T10:25:40Z)
• Knowledge Diffusion for Distillation [53.908314960324915]
  The representation gap between teacher and student is an emerging topic in knowledge distillation (KD) We state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature. We propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models.
  arXiv  Detail & Related papers  (2023-05-25T04:49:34Z)
• Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation [66.25738680429463]
  Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. We propose inconsistent knowledge distillation (IKD) which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. Our method outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors.
  arXiv  Detail & Related papers  (2022-09-20T16:36:28Z)
• Towards Efficient 3D Object Detection with Knowledge Distillation [38.89710768280703]
  We explore the potential of knowledge distillation for developing efficient 3D object detectors. Our best performing model achieves $65.75%$ 2 mAPH, surpassing its teacher model and requiring only $44%$ of teacher flops. Our most efficient model runs 51 FPS on an NVIDIA A100, which is $2.2times$ faster than PointPillar with even higher accuracy.
  arXiv  Detail & Related papers  (2022-05-30T15:02:16Z)
• Prediction-Guided Distillation for Dense Object Detection [7.5320132424481505]
  We show that only a very small fraction of features within a ground-truth bounding box are responsible for a teacher's high detection performance. We propose Prediction-Guided Distillation (PGD), which focuses distillation on these key predictive regions of the teacher. Our proposed approach outperforms current state-of-the-art KD baselines on a variety of advanced one-stage detection architectures.
  arXiv  Detail & Related papers  (2022-03-10T16:46:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.