Zero-shot detection of buildings in mobile LiDAR using Language Vision Model

• URL: http://arxiv.org/abs/2404.09931v1
• Date: Mon, 15 Apr 2024 16:56:58 GMT
• Title: Zero-shot detection of buildings in mobile LiDAR using Language Vision Model
• Authors: June Moh Goo, Zichao Zeng, Jan Boehm,
• Abstract summary: Language Vision Models (LVMs) surpass the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision tasks. LVMs face significant challenges when it comes to point clouds, a representative format for representing 3D data. Our research aims to 1) apply the Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2) experiment with synthetic data to evaluate its effectiveness.
• Score: 0.8192907805418583
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances have demonstrated that Language Vision Models (LVMs) surpass the existing State-of-the-Art (SOTA) in two-dimensional (2D) computer vision tasks, motivating attempts to apply LVMs to three-dimensional (3D) data. While LVMs are efficient and effective in addressing various downstream 2D vision tasks without training, they face significant challenges when it comes to point clouds, a representative format for representing 3D data. It is more difficult to extract features from 3D data and there are challenges due to large data sizes and the cost of the collection and labelling, resulting in a notably limited availability of datasets. Moreover, constructing LVMs for point clouds is even more challenging due to the requirements for large amounts of data and training time. To address these issues, our research aims to 1) apply the Grounded SAM through Spherical Projection to transfer 3D to 2D, and 2) experiment with synthetic data to evaluate its effectiveness in bridging the gap between synthetic and real-world data domains. Our approach exhibited high performance with an accuracy of 0.96, an IoU of 0.85, precision of 0.92, recall of 0.91, and an F1 score of 0.92, confirming its potential. However, challenges such as occlusion problems and pixel-level overlaps of multi-label points during spherical image generation remain to be addressed in future studies.

Related papers

• LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving [52.83707400688378]
  LargeAD is a versatile and scalable framework designed for large-scale 3D pretraining across diverse real-world driving datasets. Our framework leverages VFMs to extract semantically rich superpixels from 2D images, which are aligned with LiDAR point clouds to generate high-quality contrastive samples. Our approach delivers significant performance improvements over state-of-the-art methods in both linear probing and fine-tuning tasks for both LiDAR-based segmentation and object detection.
  arXiv  Detail & Related papers  (2025-01-07T18:59:59Z)
• Transfer Learning from Simulated to Real Scenes for Monocular 3D Object Detection [9.708971995966476]
  This paper introduces a two-stage training strategy to address these challenges. Our approach initially trains a model on the large-scale synthetic dataset, RoadSense3D. We fine-tune the model on a combination of real-world datasets to enhance its adaptability to practical conditions.
  arXiv  Detail & Related papers  (2024-08-28T08:44:58Z)
• 4D Contrastive Superflows are Dense 3D Representation Learners [62.433137130087445]
  We introduce SuperFlow, a novel framework designed to harness consecutive LiDAR-camera pairs for establishing pretraining objectives. To further boost learning efficiency, we incorporate a plug-and-play view consistency module that enhances alignment of the knowledge distilled from camera views.
  arXiv  Detail & Related papers  (2024-07-08T17:59:54Z)
• Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes [65.22070581594426]
  "Implicit-Zoo" is a large-scale dataset requiring thousands of GPU training days to facilitate research and development in this field. We showcase two immediate benefits as it enables to: (1) learn token locations for transformer models; (2) directly regress 3D cameras poses of 2D images with respect to NeRF models. This in turn leads to an improved performance in all three task of image classification, semantic segmentation, and 3D pose regression, thereby unlocking new avenues for research.
  arXiv  Detail & Related papers  (2024-06-25T10:20:44Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• FILP-3D: Enhancing 3D Few-shot Class-incremental Learning with Pre-trained Vision-Language Models [59.13757801286343]
  Few-shot class-incremental learning aims to mitigate the catastrophic forgetting issue when a model is incrementally trained on limited data. We introduce the FILP-3D framework with two novel components: the Redundant Feature Eliminator (RFE) for feature space misalignment and the Spatial Noise Compensator (SNC) for significant noise.
  arXiv  Detail & Related papers  (2023-12-28T14:52:07Z)
• Deep Generative Models on 3D Representations: A Survey [81.73385191402419]
  Generative models aim to learn the distribution of observed data by generating new instances. Recently, researchers started to shift focus from 2D to 3D space. representing 3D data poses significantly greater challenges.
  arXiv  Detail & Related papers  (2022-10-27T17:59:50Z)
• Efficient Urban-scale Point Clouds Segmentation with BEV Projection [0.0]
  Most deep point clouds models directly conduct learning on 3D point clouds. We propose to transfer the 3D point clouds to dense bird's-eye-view projection.
  arXiv  Detail & Related papers  (2021-09-19T06:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.