SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillation

• URL: http://arxiv.org/abs/2506.17290v1
• Date: Mon, 16 Jun 2025 07:32:58 GMT
• Title: SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillation
• Authors: Yuqi Li, Junhao Dong, Zeyu Dong, Chuanguang Yang, Zhulin An, Yongjun Xu,
• Abstract summary: 3D point cloud segmentation faces practical challenges due to the computational complexity and deployment limitations of large-scale transformer-based models.<n>We propose a novel Structure- and Relation-aware Knowledge Distillation framework, named SRKD, that transfers rich geometric and semantic knowledge from a large frozen teacher model to a lightweight student model.<n>Our method achieves significantly reduced model complexity, demonstrating its effectiveness and efficiency in real-world deployment scenarios.
• Score: 25.38025028623991
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D point cloud segmentation faces practical challenges due to the computational complexity and deployment limitations of large-scale transformer-based models. To address this, we propose a novel Structure- and Relation-aware Knowledge Distillation framework, named SRKD, that transfers rich geometric and semantic knowledge from a large frozen teacher model (>100M) to a lightweight student model (<15M). Specifically, we propose an affinity matrix-based relation alignment module, which distills structural dependencies from the teacher to the student through point-wise similarity matching, enhancing the student's capability to learn contextual interactions. Meanwhile, we introduce a cross-sample mini-batch construction strategy that enables the student to perceive stable and generalized geometric structure. This aligns across diverse point cloud instances of the teacher, rather than within a single sample. Additionally, KL divergence is applied to align semantic distributions, and ground-truth supervision further reinforces accurate segmentation. Our method achieves state of the art performance with significantly reduced model complexity, demonstrating its effectiveness and efficiency in real-world deployment scenarios. Our Code is available at https://github.com/itsnotacie/SRKD.

Related papers

• Structural Similarity-Inspired Unfolding for Lightweight Image Super-Resolution [88.20464308588889]
  We propose a Structural Similarity-Inspired Unfolding (SSIU) method for efficient image SR.<n>This method is designed through unfolding an SR optimization function constrained by structural similarity.<n>Our model outperforms current state-of-the-art models, boasting lower parameter counts and reduced memory consumption.
  arXiv  Detail & Related papers  (2025-06-13T14:29:40Z)
• Intra-class Patch Swap for Self-Distillation [3.282914142012984]
  We propose a teacher-free distillation framework based on a single student network.<n>Our approach is built on a simple yet highly effective augmentation, called intra-class patch swap augmentation.<n>Our method consistently outperforms both existing self-distillation baselines and conventional teacher-based KD approaches.
  arXiv  Detail & Related papers  (2025-05-20T09:30:19Z)
• Structured Agent Distillation for Large Language Model [58.22497891295258]
  We propose Structured Agent Distillation, a framework that compresses large LLM-based agents into smaller student models.<n>Our method segments trajectories into [REASON] and [ACT] spans, applying segment-specific losses to align each component with the teacher's behavior.<n>Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines.
  arXiv  Detail & Related papers  (2025-05-20T02:01:55Z)
• TAS: Distilling Arbitrary Teacher and Student via a Hybrid Assistant [52.0297393822012]
  We introduce an assistant model as a bridge to facilitate smooth feature knowledge transfer between heterogeneous teachers and students. Within our proposed design principle, the assistant model combines the advantages of cross-architecture inductive biases and module functions. Our proposed method is evaluated across some homogeneous model pairs and arbitrary heterogeneous combinations of CNNs, ViTs, spatial KDs.
  arXiv  Detail & Related papers  (2024-10-16T08:02:49Z)
• Exploring Graph-based Knowledge: Multi-Level Feature Distillation via Channels Relational Graph [8.646512035461994]
  In visual tasks, large teacher models capture essential features and deep information, enhancing performance. We propose a distillation framework based on graph knowledge, including a multi-level feature alignment strategy. We emphasize spectral embedding (SE) as a key technique in our distillation process, which merges the student's feature space with the relational knowledge and structural complexities similar to the teacher network.
  arXiv  Detail & Related papers  (2024-05-14T12:37:05Z)
• One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation [69.65734716679925]
  Knowledge distillation has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. Most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family. We propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures.
  arXiv  Detail & Related papers  (2023-10-30T11:13:02Z)
• EmbedDistill: A Geometric Knowledge Distillation for Information Retrieval [83.79667141681418]
  Large neural models (such as Transformers) achieve state-of-the-art performance for information retrieval (IR) We propose a novel distillation approach that leverages the relative geometry among queries and documents learned by the large teacher model. We show that our approach successfully distills from both dual-encoder (DE) and cross-encoder (CE) teacher models to 1/10th size asymmetric students that can retain 95-97% of the teacher performance.
  arXiv  Detail & Related papers  (2023-01-27T22:04:37Z)
• Structural Knowledge Distillation for Object Detection [2.690687471762886]
  We propose a replacement for the pixel-wise independent lp-norm based on the structural similarity (SSIM) Our method adds only little computational overhead, is straightforward to implement and at the same time it significantly outperforms the standard lp-norms.
  arXiv  Detail & Related papers  (2022-11-23T17:06:52Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.