OpenPatch: a 3D patchwork for Out-Of-Distribution detection

• URL: http://arxiv.org/abs/2310.03388v3
• Date: Mon, 23 Oct 2023 22:11:02 GMT
• Title: OpenPatch: a 3D patchwork for Out-Of-Distribution detection
• Authors: Paolo Rabino, Antonio Alliegro, Francesco Cappio Borlino, Tatiana Tommasi
• Abstract summary: We present an approach for the task of semantic novelty detection on real-world point cloud samples when the reference known data are synthetic. OpenPatch builds on a large pre-trained model and simply extracts from its intermediate features a set of patch representations that describe each known class. We demonstrate that OpenPatch excels in both the full and few-shot known sample scenarios.
• Score: 16.262921993755892
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Moving deep learning models from the laboratory setting to the open world entails preparing them to handle unforeseen conditions. In several applications the occurrence of novel classes during deployment poses a significant threat, thus it is crucial to effectively detect them. Ideally, this skill should be used when needed without requiring any further computational training effort at every new task. Out-of-distribution detection has attracted significant attention in the last years, however the majority of the studies deal with 2D images ignoring the inherent 3D nature of the real-world and often confusing between domain and semantic novelty. In this work, we focus on the latter, considering the objects geometric structure captured by 3D point clouds regardless of the specific domain. We advance the field by introducing OpenPatch that builds on a large pre-trained model and simply extracts from its intermediate features a set of patch representations that describe each known class. For any new sample, we obtain a novelty score by evaluating whether it can be recomposed mainly by patches of a single known class or rather via the contribution of multiple classes. We present an extensive experimental evaluation of our approach for the task of semantic novelty detection on real-world point cloud samples when the reference known data are synthetic. We demonstrate that OpenPatch excels in both the full and few-shot known sample scenarios, showcasing its robustness across varying pre-training objectives and network backbones. The inherent training-free nature of our method allows for its immediate application to a wide array of real-world tasks, offering a compelling advantage over approaches that need expensive retraining efforts.

Related papers

• Find n' Propagate: Open-Vocabulary 3D Object Detection in Urban Environments [67.83787474506073]
  We tackle the limitations of current LiDAR-based 3D object detection systems. We introduce a universal textscFind n' Propagate approach for 3D OV tasks. We achieve up to a 3.97-fold increase in Average Precision (AP) for novel object classes.
  arXiv  Detail & Related papers  (2024-03-20T12:51:30Z)
• Generalized Label-Efficient 3D Scene Parsing via Hierarchical Feature Aligned Pre-Training and Region-Aware Fine-tuning [55.517000360348725]
  This work presents a framework for dealing with 3D scene understanding when the labeled scenes are quite limited. To extract knowledge for novel categories from the pre-trained vision-language models, we propose a hierarchical feature-aligned pre-training and knowledge distillation strategy. Experiments with both indoor and outdoor scenes demonstrated the effectiveness of our approach in both data-efficient learning and open-world few-shot learning.
  arXiv  Detail & Related papers  (2023-12-01T15:47:04Z)
• Open-CRB: Towards Open World Active Learning for 3D Object Detection [40.80953254074535]
  LiDAR-based 3D object detection has recently seen significant advancements through active learning (AL) In real-world deployments where streaming point clouds may include unknown or novel objects, the ability of current AL methods to capture such objects remains unexplored. This paper investigates a more practical and challenging research task: Open World Active Learning for 3D Object Detection (OWAL-3D)
  arXiv  Detail & Related papers  (2023-10-16T13:32:53Z)
• Generalized Few-Shot 3D Object Detection of LiDAR Point Cloud for Autonomous Driving [91.39625612027386]
  We propose a novel task, called generalized few-shot 3D object detection, where we have a large amount of training data for common (base) objects, but only a few data for rare (novel) classes. Specifically, we analyze in-depth differences between images and point clouds, and then present a practical principle for the few-shot setting in the 3D LiDAR dataset. To solve this task, we propose an incremental fine-tuning method to extend existing 3D detection models to recognize both common and rare objects.
  arXiv  Detail & Related papers  (2023-02-08T07:11:36Z)
• ALSO: Automotive Lidar Self-supervision by Occupancy estimation [70.70557577874155]
  We propose a new self-supervised method for pre-training the backbone of deep perception models operating on point clouds. The core idea is to train the model on a pretext task which is the reconstruction of the surface on which the 3D points are sampled. The intuition is that if the network is able to reconstruct the scene surface, given only sparse input points, then it probably also captures some fragments of semantic information.
  arXiv  Detail & Related papers  (2022-12-12T13:10:19Z)
• Towards Open Set 3D Learning: A Benchmark on Object Point Clouds [17.145309633743747]
  This paper provides the first broad study on Open Set 3D learning. We introduce a novel testbed with settings of increasing difficulty in terms of category semantic shift. We investigate the related out-of-distribution and Open Set 2D literature to understand if and how their most recent approaches are effective on 3D data.
  arXiv  Detail & Related papers  (2022-07-23T17:00:45Z)
• NovelCraft: A Dataset for Novelty Detection and Discovery in Open Worlds [14.265615838391703]
  NovelCraft dataset contains episodic data of the images and symbolic world-states seen by an agent completing a pogo stick assembly task within a modified Minecraft environment. Our visual novelty detection benchmark finds that methods that rank best on popular area-under-the-curve metrics may be outperformed by simpler alternatives. Further multimodal novelty detection experiments suggest that methods that fuse both visual and symbolic information can improve time until detection as well as overall discrimination.
  arXiv  Detail & Related papers  (2022-06-23T14:31:33Z)
• RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection [138.2892824662943]
  A promising solution is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. Recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective.
  arXiv  Detail & Related papers  (2021-08-17T17:56:12Z)
• Few-shot Weakly-Supervised Object Detection via Directional Statistics [55.97230224399744]
  We propose a probabilistic multiple instance learning approach for few-shot Common Object Localization (COL) and few-shot Weakly Supervised Object Detection (WSOD) Our model simultaneously learns the distribution of the novel objects and localizes them via expectation-maximization steps. Our experiments show that the proposed method, despite being simple, outperforms strong baselines in few-shot COL and WSOD, as well as large-scale WSOD tasks.
  arXiv  Detail & Related papers  (2021-03-25T22:34:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.