Zero-shot detection of buildings in mobile LiDAR using Language Vision Model

• URL: http://arxiv.org/abs/2404.09931v1
• Date: Mon, 15 Apr 2024 16:56:58 GMT
• Title: Zero-shot detection of buildings in mobile LiDAR using Language Vision Model
• Authors: June Moh Goo, Zichao Zeng, Jan Boehm,
• Abstract summary: Language Vision Models (LVMs) surpass the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision tasks. LVMs face significant challenges when it comes to point clouds, a representative format for representing 3D data. Our research aims to 1) apply the Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2) experiment with synthetic data to evaluate its effectiveness.
• Score: 0.8192907805418583
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances have demonstrated that Language Vision Models (LVMs) surpass the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision tasks, motivating attempts to apply LVMs to three-dimensional (3D) data. While LVMs are efficient and effective in addressing various downstream 2D vision tasks without training, they face significant challenges when it comes to point clouds, a representative format for representing 3D data. It is more difficult to extract features from 3D data and there are challenges due to large data sizes and the cost of the collection and labelling, resulting in a notably limited availability of datasets. Moreover, constructing LVMs for point clouds is even more challenging due to the requirements for large amounts of data and training time. To address these issues, our research aims to 1) apply the Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2) experiment with synthetic data to evaluate its effectiveness in bridging the gap between synthetic and real-world data domains. Our approach exhibited high performance with an accuracy of 0.96, an IoU of 0.85, precision of 0.92, recall of 0.91, and an F1 score of 0.92, confirming its potential. However, challenges such as occlusion problems and pixel-level overlaps of multi-label points during spherical image generation remain to be addressed in future studies.

Related papers

• Transfer Learning from Simulated to Real Scenes for Monocular 3D Object Detection [9.708971995966476]
  This paper introduces a two-stage training strategy to address these challenges. Our approach initially trains a model on the large-scale synthetic dataset, RoadSense3D. We fine-tune the model on a combination of real-world datasets to enhance its adaptability to practical conditions.
  arXiv  Detail & Related papers  (2024-08-28T08:44:58Z)
• 4D Contrastive Superflows are Dense 3D Representation Learners [62.433137130087445]
  We introduce SuperFlow, a novel framework designed to harness consecutive LiDAR-camera pairs for establishing pretraining objectives. To further boost learning efficiency, we incorporate a plug-and-play view consistency module that enhances alignment of the knowledge distilled from camera views.
  arXiv  Detail & Related papers  (2024-07-08T17:59:54Z)
• Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes [65.22070581594426]
  "Implicit-Zoo" is a large-scale dataset requiring thousands of GPU training days to facilitate research and development in this field. We showcase two immediate benefits as it enables to: (1) learn token locations for transformer models; (2) directly regress 3D cameras poses of 2D images with respect to NeRF models. This in turn leads to an improved performance in all three task of image classification, semantic segmentation, and 3D pose regression, thereby unlocking new avenues for research.
  arXiv  Detail & Related papers  (2024-06-25T10:20:44Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained Vision-Language Models [62.663113296987085]
  Few-shot class-incremental learning aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. We introduce two novel components: the Redundant Feature Eliminator (RFE) and the Spatial Noise Compensator (SNC) Considering the imbalance in existing 3D datasets, we also propose new evaluation metrics that offer a more nuanced assessment of a 3D FSCIL model.
  arXiv  Detail & Related papers  (2023-12-28T14:52:07Z)
• Leveraging Large-Scale Pretrained Vision Foundation Models for Label-Efficient 3D Point Cloud Segmentation [67.07112533415116]
  We present a novel framework that adapts various foundational models for the 3D point cloud segmentation task. Our approach involves making initial predictions of 2D semantic masks using different large vision models. To generate robust 3D semantic pseudo labels, we introduce a semantic label fusion strategy that effectively combines all the results via voting.
  arXiv  Detail & Related papers  (2023-11-03T15:41:15Z)
• Deep Generative Models on 3D Representations: A Survey [81.73385191402419]
  Generative models aim to learn the distribution of observed data by generating new instances. Recently, researchers started to shift focus from 2D to 3D space. representing 3D data poses significantly greater challenges.
  arXiv  Detail & Related papers  (2022-10-27T17:59:50Z)
• Efficient Urban-scale Point Clouds Segmentation with BEV Projection [0.0]
  Most deep point clouds models directly conduct learning on 3D point clouds. We propose to transfer the 3D point clouds to dense bird's-eye-view projection.
  arXiv  Detail & Related papers  (2021-09-19T06:49:59Z)
• Learning to Drop Points for LiDAR Scan Synthesis [5.132259673802809]
  Generative modeling of 3D scenes is a crucial topic for aiding mobile robots to improve unreliable observations. Most existing studies on point clouds have focused on small and uniform-density data. 3D LiDAR point clouds widely used in mobile robots are non-trivial to be handled because of the large number of points and varying-density. This paper proposes a novel framework based on generative adversarial networks to synthesize realistic LiDAR data as an improved 2D representation.
  arXiv  Detail & Related papers  (2021-02-23T21:53:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.