Learning Robust 3D Representation from CLIP via Dual Denoising

• URL: http://arxiv.org/abs/2407.00905v1
• Date: Mon, 1 Jul 2024 02:15:03 GMT
• Title: Learning Robust 3D Representation from CLIP via Dual Denoising
• Authors: Shuqing Luo, Bowen Qu, Wei Gao,
• Abstract summary: We propose Dual Denoising, a novel framework for learning robust and well-generalized 3D representations from CLIP. It combines a denoising-based proxy task with a novel feature denoising network for 3D pre-training. Experiments show that our model can effectively improve the representation learning performance and adversarial robustness of the 3D learning network.
• Score: 4.230780744307392
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we explore a critical yet under-investigated issue: how to learn robust and well-generalized 3D representation from pre-trained vision language models such as CLIP. Previous works have demonstrated that cross-modal distillation can provide rich and useful knowledge for 3D data. However, like most deep learning models, the resultant 3D learning network is still vulnerable to adversarial attacks especially the iterative attack. In this work, we propose Dual Denoising, a novel framework for learning robust and well-generalized 3D representations from CLIP. It combines a denoising-based proxy task with a novel feature denoising network for 3D pre-training. Additionally, we propose utilizing parallel noise inference to enhance the generalization of point cloud features under cross domain settings. Experiments show that our model can effectively improve the representation learning performance and adversarial robustness of the 3D learning network under zero-shot settings without adversarial training. Our code is available at https://github.com/luoshuqing2001/Dual_Denoising.

Related papers

• Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels [69.55622471172941]
  Large-scale vision 2D vision language models, such as CLIP can be aligned with a 3D encoder to learn generalizable (open-vocabulary) 3D vision models. We propose an optimization framework: Cross-MoST: Cross-Modal Self-Training, to improve the label-free classification performance of a zero-shot 3D vision model.
  arXiv  Detail & Related papers  (2024-04-15T21:30:50Z)
• FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained Vision-Language Models [62.663113296987085]
  Few-shot class-incremental learning aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. We introduce two novel components: the Redundant Feature Eliminator (RFE) and the Spatial Noise Compensator (SNC) Considering the imbalance in existing 3D datasets, we also propose new evaluation metrics that offer a more nuanced assessment of a 3D FSCIL model.
  arXiv  Detail & Related papers  (2023-12-28T14:52:07Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [114.47216525866435]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation. For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning [37.155772047656114]
  Masked Scene Contrast (MSC) framework is capable of extracting comprehensive 3D representations more efficiently and effectively. MSC also enables large-scale 3D pre-training across multiple datasets.
  arXiv  Detail & Related papers  (2023-03-24T17:59:58Z)
• CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP [55.864132158596206]
  Contrastive Language-Image Pre-training (CLIP) achieves promising results in 2D zero-shot and few-shot learning. We make the first attempt to investigate how CLIP knowledge benefits 3D scene understanding. We propose CLIP2Scene, a framework that transfers CLIP knowledge from 2D image-text pre-trained models to a 3D point cloud network.
  arXiv  Detail & Related papers  (2023-01-12T10:42:39Z)
• ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding [110.07170245531464]
  Current 3D models are limited by datasets with a small number of annotated data and a pre-defined set of categories. Recent advances have shown that similar problems can be significantly alleviated by employing knowledge from other modalities, such as language. We learn a unified representation of images, texts, and 3D point clouds by pre-training with object triplets from the three modalities.
  arXiv  Detail & Related papers  (2022-12-10T01:34:47Z)
• PointACL:Adversarial Contrastive Learning for Robust Point Clouds Representation under Adversarial Attack [73.3371797787823]
  Adversarial contrastive learning (ACL) is considered an effective way to improve the robustness of pre-trained models. We present our robust aware loss function to train self-supervised contrastive learning framework adversarially. We validate our method, PointACL on downstream tasks, including 3D classification and 3D segmentation with multiple datasets.
  arXiv  Detail & Related papers  (2022-09-14T22:58:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.