Concept-based Explainable Data Mining with VLM for 3D Detection

• URL: http://arxiv.org/abs/2512.05482v1
• Date: Fri, 05 Dec 2025 07:18:45 GMT
• Title: Concept-based Explainable Data Mining with VLM for 3D Detection
• Authors: Mai Tsujimoto,
• Abstract summary: This paper proposes a novel cross-modal framework that leverages 2D Vision-Language Models to identify and mine rare objects from driving scenes.<n>Our approach synthesizes complementary techniques such as object detection, semantic feature extraction, dimensionality reduction, and multi-faceted outlier detection.<n> Experiments on the nuScenes dataset demonstrate that this concept-guided data mining strategy enhances the performance of 3D object detection models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Rare-object detection remains a challenging task in autonomous driving systems, particularly when relying solely on point cloud data. Although Vision-Language Models (VLMs) exhibit strong capabilities in image understanding, their potential to enhance 3D object detection through intelligent data mining has not been fully explored. This paper proposes a novel cross-modal framework that leverages 2D VLMs to identify and mine rare objects from driving scenes, thereby improving 3D object detection performance. Our approach synthesizes complementary techniques such as object detection, semantic feature extraction, dimensionality reduction, and multi-faceted outlier detection into a cohesive, explainable pipeline that systematically identifies rare but critical objects in driving scenes. By combining Isolation Forest and t-SNE-based outlier detection methods with concept-based filtering, the framework effectively identifies semantically meaningful rare objects. A key strength of this approach lies in its ability to extract and annotate targeted rare object concepts such as construction vehicles, motorcycles, and barriers. This substantially reduces the annotation burden and focuses only on the most valuable training samples. Experiments on the nuScenes dataset demonstrate that this concept-guided data mining strategy enhances the performance of 3D object detection models while utilizing only a fraction of the training data, with particularly notable improvements for challenging object categories such as trailers and bicycles compared with the same amount of random data. This finding has substantial implications for the efficient curation of datasets in safety-critical autonomous systems.

Related papers

• HD-OOD3D: Supervised and Unsupervised Out-of-Distribution object detection in LiDAR data [6.372261626436676]
  HD-OOD3D is a novel two-stage method for detecting unknown objects.<n>To address the challenge of scaling the learning of unknown objects, we explore unsupervised training strategies to generate pseudo-labels for unknowns.
  arXiv  Detail & Related papers  (2024-10-31T09:29:55Z)
• Label-Efficient 3D Object Detection For Road-Side Units [10.663986706501188]
  Collaborative perception can enhance the perception of autonomous vehicles via deep information fusion with intelligent roadside units (RSU) The data-hungry nature of these methods creates a major hurdle for their real-world deployment, particularly due to the need for annotated RSU data. We devise a label-efficient object detection method for RSU based on unsupervised object discovery.
  arXiv  Detail & Related papers  (2024-04-09T12:29:16Z)
• Cross-Cluster Shifting for Efficient and Effective 3D Object Detection in Autonomous Driving [69.20604395205248]
  We present a new 3D point-based detector model, named Shift-SSD, for precise 3D object detection in autonomous driving. We introduce an intriguing Cross-Cluster Shifting operation to unleash the representation capacity of the point-based detector. We conduct extensive experiments on the KITTI, runtime, and nuScenes datasets, and the results demonstrate the state-of-the-art performance of Shift-SSD.
  arXiv  Detail & Related papers  (2024-03-10T10:36:32Z)
• LS-VOS: Identifying Outliers in 3D Object Detections Using Latent Space Virtual Outlier Synthesis [10.920640666237833]
  LiDAR-based 3D object detectors have achieved unprecedented speed and accuracy in autonomous driving applications. They are often biased toward high-confidence predictions or return detections where no real object is present. We propose LS-VOS, a framework for identifying outliers in 3D object detections.
  arXiv  Detail & Related papers  (2023-10-02T07:44:26Z)
• PatchContrast: Self-Supervised Pre-training for 3D Object Detection [14.493213289990962]
  We introduce PatchContrast, a novel self-supervised point cloud pre-training framework for 3D object detection.<n>We show that our method outperforms existing state-of-the-art models on three commonly-used 3D detection datasets.
  arXiv  Detail & Related papers  (2023-08-14T07:45:54Z)
• Generalized Few-Shot 3D Object Detection of LiDAR Point Cloud for Autonomous Driving [91.39625612027386]
  We propose a novel task, called generalized few-shot 3D object detection, where we have a large amount of training data for common (base) objects, but only a few data for rare (novel) classes. Specifically, we analyze in-depth differences between images and point clouds, and then present a practical principle for the few-shot setting in the 3D LiDAR dataset. To solve this task, we propose an incremental fine-tuning method to extend existing 3D detection models to recognize both common and rare objects.
  arXiv  Detail & Related papers  (2023-02-08T07:11:36Z)
• AGO-Net: Association-Guided 3D Point Cloud Object Detection Network [86.10213302724085]
  We propose a novel 3D detection framework that associates intact features for objects via domain adaptation. We achieve new state-of-the-art performance on the KITTI 3D detection benchmark in both accuracy and speed.
  arXiv  Detail & Related papers  (2022-08-24T16:54:38Z)
• The Devil is in the Task: Exploiting Reciprocal Appearance-Localization Features for Monocular 3D Object Detection [62.1185839286255]
  Low-cost monocular 3D object detection plays a fundamental role in autonomous driving. We introduce a Dynamic Feature Reflecting Network, named DFR-Net. We rank 1st among all the monocular 3D object detectors in the KITTI test set.
  arXiv  Detail & Related papers  (2021-12-28T07:31:18Z)
• Weakly Supervised Training of Monocular 3D Object Detectors Using Wide Baseline Multi-view Traffic Camera Data [19.63193201107591]
  7DoF prediction of vehicles at an intersection is an important task for assessing potential conflicts between road users. We develop an approach using a weakly supervised method of fine tuning 3D object detectors for traffic observation cameras. Our method achieves vehicle 7DoF pose prediction accuracy on our dataset comparable to the top performing monocular 3D object detectors on autonomous vehicle datasets.
  arXiv  Detail & Related papers  (2021-10-21T08:26:48Z)
• Learnable Online Graph Representations for 3D Multi-Object Tracking [156.58876381318402]
  We propose a unified and learning based approach to the 3D MOT problem. We employ a Neural Message Passing network for data association that is fully trainable. We show the merit of the proposed approach on the publicly available nuScenes dataset by achieving state-of-the-art performance of 65.6% AMOTA and 58% fewer ID-switches.
  arXiv  Detail & Related papers  (2021-04-23T17:59:28Z)
• SESS: Self-Ensembling Semi-Supervised 3D Object Detection [138.80825169240302]
  We propose SESS, a self-ensembling semi-supervised 3D object detection framework. Specifically, we design a thorough perturbation scheme to enhance generalization of the network on unlabeled and new unseen data. Our SESS achieves competitive performance compared to the state-of-the-art fully-supervised method by using only 50% labeled data.
  arXiv  Detail & Related papers  (2019-12-26T08:48:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.