PMT-MAE: Dual-Branch Self-Supervised Learning with Distillation for Efficient Point Cloud Classification

• URL: http://arxiv.org/abs/2409.02007v2
• Date: Mon, 16 Sep 2024 16:51:50 GMT
• Title: PMT-MAE: Dual-Branch Self-Supervised Learning with Distillation for Efficient Point Cloud Classification
• Authors: Qiang Zheng, Chao Zhang, Jian Sun,
• Abstract summary: This paper introduces PMT-MAE, a novel self-supervised learning framework for point cloud classification. PMT-MAE features a dual-branch architecture that integrates Transformer and components to capture rich features. PMT-MAE surpasses the baseline Point-MAE (93.2%) and the teacher Point-M2AE (93.4%), underscoring its ability to learn discnative 3D point cloud representations.
• Score: 46.266960248570086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Advances in self-supervised learning are essential for enhancing feature extraction and understanding in point cloud processing. This paper introduces PMT-MAE (Point MLP-Transformer Masked Autoencoder), a novel self-supervised learning framework for point cloud classification. PMT-MAE features a dual-branch architecture that integrates Transformer and MLP components to capture rich features. The Transformer branch leverages global self-attention for intricate feature interactions, while the parallel MLP branch processes tokens through shared fully connected layers, offering a complementary feature transformation pathway. A fusion mechanism then combines these features, enhancing the model's capacity to learn comprehensive 3D representations. Guided by the sophisticated teacher model Point-M2AE, PMT-MAE employs a distillation strategy that includes feature distillation during pre-training and logit distillation during fine-tuning, ensuring effective knowledge transfer. On the ModelNet40 classification task, achieving an accuracy of 93.6\% without employing voting strategy, PMT-MAE surpasses the baseline Point-MAE (93.2\%) and the teacher Point-M2AE (93.4\%), underscoring its ability to learn discriminative 3D point cloud representations. Additionally, this framework demonstrates high efficiency, requiring only 40 epochs for both pre-training and fine-tuning. PMT-MAE's effectiveness and efficiency render it well-suited for scenarios with limited computational resources, positioning it as a promising solution for practical point cloud analysis.

Related papers

• PointLoRA: Low-Rank Adaptation with Token Selection for Point Cloud Learning [54.99373314906667]
  Self-supervised representation learning for point cloud has demonstrated effectiveness in improving pre-trained model performance across diverse tasks. As pre-trained models grow in complexity, fully fine-tuning them for downstream applications demands substantial computational and storage resources. We propose PointLoRA, a simple yet effective method that combines low-rank adaptation (LoRA) with multi-scale token selection to efficiently fine-tune point cloud models.
  arXiv  Detail & Related papers  (2025-04-22T16:41:21Z)
• MAAM: A Lightweight Multi-Agent Aggregation Module for Efficient Image Classification Based on the MindSpore Framework [4.307728769243765]
  We propose a lightweight attention architecture integrated with the MindSpore framework. The Multi-Agent Aggregation Module (MAAM) employs three parallel agent branches with independently parameterized operations to extract heterogeneous features. Using MindSpore's dynamic computational graph and operator fusion, MAAM achieves 87.0% accuracy on the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2025-04-18T09:19:07Z)
• Unleashing the Potential of Mamba: Boosting a LiDAR 3D Sparse Detector by Using Cross-Model Knowledge Distillation [22.653014803666668]
  We propose a Faster LiDAR 3D object detection framework, called FASD, which implements heterogeneous model distillation by adaptively uniform cross-model voxel features. We aim to distill the transformer's capacity for high-performance sequence modeling into Mamba models with low FLOPs, achieving a significant improvement in accuracy through knowledge transfer. We evaluated the framework on datasets and nuScenes, achieving a 4x reduction in resource consumption and a 1-2% performance improvement over the current SoTA methods.
  arXiv  Detail & Related papers  (2024-09-17T09:30:43Z)
• PointMT: Efficient Point Cloud Analysis with Hybrid MLP-Transformer Architecture [46.266960248570086]
  This study tackles the quadratic complexity of the self-attention mechanism by introducing a complexity local attention mechanism for effective feature aggregation. We also introduce a parameter-free channel temperature adaptation mechanism that adaptively adjusts the attention weight distribution in each channel. We show that PointMT achieves performance comparable to state-of-the-art methods while maintaining an optimal balance between performance and accuracy.
  arXiv  Detail & Related papers  (2024-08-10T10:16:03Z)
• Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning [116.75939193785143]
  Contrastive learning (CL) for Vision Transformers (ViTs) in image domains has achieved performance comparable to CL for traditional convolutional backbones. In 3D point cloud pretraining with ViTs, masked autoencoder (MAE) modeling remains dominant.
  arXiv  Detail & Related papers  (2024-07-08T12:28:56Z)
• Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation [56.053397775016755]
  We propose a sequential approach to knowledge distillation that progressively transfers the knowledge of a set of teacher detectors to a given lightweight student. To the best of our knowledge, we are the first to successfully distill knowledge from Transformer-based teacher detectors to convolution-based students.
  arXiv  Detail & Related papers  (2023-08-17T17:17:08Z)
• CLR-GAM: Contrastive Point Cloud Learning with Guided Augmentation and Feature Mapping [12.679625717350113]
  We present CLR-GAM, a contrastive learning-based framework with Guided Augmentation (GA) for efficient dynamic exploration strategy. We empirically demonstrate that the proposed approach achieves state-of-the-art performance on both simulated and real-world 3D point cloud datasets.
  arXiv  Detail & Related papers  (2023-02-28T04:38:52Z)
• Magic ELF: Image Deraining Meets Association Learning and Transformer [63.761812092934576]
  This paper aims to unify CNN and Transformer to take advantage of their learning merits for image deraining. A novel multi-input attention module (MAM) is proposed to associate rain removal and background recovery. Our proposed method (dubbed as ELF) outperforms the state-of-the-art approach (MPRNet) by 0.25 dB on average.
  arXiv  Detail & Related papers  (2022-07-21T12:50:54Z)
• Weakly Supervised Semantic Segmentation via Alternative Self-Dual Teaching [82.71578668091914]
  This paper establishes a compact learning framework that embeds the classification and mask-refinement components into a unified deep model. We propose a novel alternative self-dual teaching (ASDT) mechanism to encourage high-quality knowledge interaction.
  arXiv  Detail & Related papers  (2021-12-17T11:56:56Z)
• Energy-Efficient and Federated Meta-Learning via Projected Stochastic Gradient Ascent [79.58680275615752]
  We propose an energy-efficient federated meta-learning framework. We assume each task is owned by a separate agent, so a limited number of tasks is used to train a meta-model.
  arXiv  Detail & Related papers  (2021-05-31T08:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.