CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation

• URL: http://arxiv.org/abs/2406.18941v1
• Date: Thu, 27 Jun 2024 07:13:09 GMT
• Title: CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation
• Authors: Zuo Zuo, Jiahao Dong, Yao Wu, Yanyun Qu, Zongze Wu,
• Abstract summary: We propose CLIP3D-AD, an efficient 3D-FSAD method extended on CLIP. We synthesize anomalous images on given normal images as sample pairs to adapt CLIP for 3D anomaly classification and segmentation. Our method has a competitive performance of 3D few-shot anomaly classification and segmentation on MVTec-3D AD dataset.
• Score: 22.850815902535988
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Few-shot anomaly detection methods can effectively address data collecting difficulty in industrial scenarios. Compared to 2D few-shot anomaly detection (2D-FSAD), 3D few-shot anomaly detection (3D-FSAD) is still an unexplored but essential task. In this paper, we propose CLIP3D-AD, an efficient 3D-FSAD method extended on CLIP. We successfully transfer strong generalization ability of CLIP into 3D-FSAD. Specifically, we synthesize anomalous images on given normal images as sample pairs to adapt CLIP for 3D anomaly classification and segmentation. For classification, we introduce an image adapter and a text adapter to fine-tune global visual features and text features. Meanwhile, we propose a coarse-to-fine decoder to fuse and facilitate intermediate multi-layer visual representations of CLIP. To benefit from geometry information of point cloud and eliminate modality and data discrepancy when processed by CLIP, we project and render point cloud to multi-view normal and anomalous images. Then we design multi-view fusion module to fuse features of multi-view images extracted by CLIP which are used to facilitate visual representations for further enhancing vision-language correlation. Extensive experiments demonstrate that our method has a competitive performance of 3D few-shot anomaly classification and segmentation on MVTec-3D AD dataset.

Related papers

• Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval [76.86914849263168]
  Open-set 3D object retrieval is an emerging task aiming to retrieve 3D objects of unseen categories beyond the training set.<n>Existing methods typically utilize all modalities (i.e., voxels, point clouds, multi-view images) and train specific backbones before fusion.<n>We present a framework named Describe, Adapt and Combine (DAC) by taking only multi-view images for open-set 3DOR.
  arXiv  Detail & Related papers  (2025-07-29T04:11:05Z)
• PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection [13.60524473223155]
  This paper introduces PointAD, a novel approach that transfers the strong generalization capabilities of CLIP for recognizing 3D anomalies on unseen objects. PointAD renders 3D anomalies into multiple 2D renderings and projects them back into 3D space. Our model can directly integrate RGB information, further enhancing the understanding of 3D anomalies in a plug-and-play manner.
  arXiv  Detail & Related papers  (2024-10-01T01:40:22Z)
• OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation [67.56268991234371]
  OV-Uni3DETR achieves the state-of-the-art performance on various scenarios, surpassing existing methods by more than 6% on average. Code and pre-trained models will be released later.
  arXiv  Detail & Related papers  (2024-03-28T17:05:04Z)
• TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding [28.112402580426174]
  TriAdapter Multi-Modal Learning (TAMM) is a novel two-stage learning approach based on three synergistic adapters. TAMM consistently enhances 3D representations for a wide range of 3D encoder architectures, pre-training datasets, and downstream tasks.
  arXiv  Detail & Related papers  (2024-02-28T17:18:38Z)
• 3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features [70.50665869806188]
  3DiffTection is a state-of-the-art method for 3D object detection from single images. We fine-tune a diffusion model to perform novel view synthesis conditioned on a single image. We further train the model on target data with detection supervision.
  arXiv  Detail & Related papers  (2023-11-07T23:46:41Z)
• Bridged Transformer for Vision and Point Cloud 3D Object Detection [92.86856146086316]
  Bridged Transformer (BrT) is an end-to-end architecture for 3D object detection. BrT learns to identify 3D and 2D object bounding boxes from both points and image patches. We experimentally show that BrT surpasses state-of-the-art methods on SUN RGB-D and ScanNetV2 datasets.
  arXiv  Detail & Related papers  (2022-10-04T05:44:22Z)
• CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training [121.46758260964114]
  Pre-training across 3D vision and language remains under development because of limited training data. Recent works attempt to transfer vision-language pre-training models to 3D vision. PointCLIP converts point cloud data to multi-view depth maps, adopting CLIP for shape classification. We propose CLIP2Point, an image-depth pre-training method by contrastive learning to transfer CLIP to the 3D domain.
  arXiv  Detail & Related papers  (2022-10-03T16:13:14Z)
• Scatter Points in Space: 3D Detection from Multi-view Monocular Images [8.71944437852952]
  3D object detection from monocular image(s) is a challenging and long-standing problem of computer vision. Recent methods tend to aggregate multiview feature by sampling regular 3D grid densely in space. We propose a learnable keypoints sampling method, which scatters pseudo surface points in 3D space, in order to keep data sparsity.
  arXiv  Detail & Related papers  (2022-08-31T09:38:05Z)
• PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition [55.38462937452363]
  We propose a unified multi-view cross-modal distillation architecture, including a pretrained deep image encoder as the teacher and a deep point encoder as the student. By pair-wise aligning multi-view visual and geometric descriptors, we can obtain more powerful deep point encoders without exhausting and complicated network modification.
  arXiv  Detail & Related papers  (2022-07-07T07:23:20Z)
• Weakly Supervised Volumetric Image Segmentation with Deformed Templates [80.04326168716493]
  We propose an approach that is truly weakly-supervised in the sense that we only need to provide a sparse set of 3D point on the surface of target objects. We will show that it outperforms a more traditional approach to weak-supervision in 3D at a reduced supervision cost.
  arXiv  Detail & Related papers  (2021-06-07T22:09:34Z)
• Object Detection on Single Monocular Images through Canonical Correlation Analysis [3.4722706398428493]
  We retrieve 3-D object information from single monocular images without using extra 3-D data like points cloud or depth images. We propose a two-dimensional CCA framework to fuse monocular images and corresponding predicted depth images for basic computer vision tasks.
  arXiv  Detail & Related papers  (2020-02-13T05:03:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.