Semantic-Drive: Democratizing Long-Tail Data Curation via Open-Vocabulary Grounding and Neuro-Symbolic VLM Consensus

• URL: http://arxiv.org/abs/2512.12012v2
• Date: Tue, 16 Dec 2025 17:15:46 GMT
• Title: Semantic-Drive: Democratizing Long-Tail Data Curation via Open-Vocabulary Grounding and Neuro-Symbolic VLM Consensus
• Abstract summary: We introduce Semantic-Drive, a local-first, neuro-symbolic framework for semantic data mining.<n>Our approach decouples perception into two stages: (1) Grounding via a real-time open-vocabulary detector (YOLOE) to anchor attention, and (2) Cognitive Analysis via a Reasoning VLM that performs forensic scene analysis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The development of robust Autonomous Vehicles (AVs) is bottlenecked by the scarcity of "Long-Tail" training data. While fleets collect petabytes of video logs, identifying rare safety-critical events (e.g., erratic jaywalking, construction diversions) remains a manual, cost-prohibitive process. Existing solutions rely on coarse metadata search, which lacks precision, or cloud-based VLMs, which are privacy-invasive and expensive. We introduce Semantic-Drive, a local-first, neuro-symbolic framework for semantic data mining. Our approach decouples perception into two stages: (1) Symbolic Grounding via a real-time open-vocabulary detector (YOLOE) to anchor attention, and (2) Cognitive Analysis via a Reasoning VLM that performs forensic scene analysis. To mitigate hallucination, we implement a "System 2" inference-time alignment strategy, utilizing a multi-model "Judge-Scout" consensus mechanism. Benchmarked on the nuScenes dataset against the Waymo Open Dataset (WOD-E2E) taxonomy, Semantic-Drive achieves a Recall of 0.966 (vs. 0.475 for CLIP) and reduces Risk Assessment Error by 40% ccompared to the best single scout models. The system runs entirely on consumer hardware (NVIDIA RTX 3090), offering a privacy-preserving alternative to the cloud.

Related papers

• SoK: Federated Learning for Intrusion Detection in Vehicular Networks [1.176565404005321]
  Modern vehicular networks face an expanding attack surface across internal Electronic Control Units (ECUs) and external Vehicle-to-Everything (V2X) communication.<n> Federated Learning (FL) has emerged as a decentralized paradigm to deploy Intrusion Detection Systems (IDS) without compromising data privacy.<n>This paper presents a Systematization of Knowledge (SoK) that unifies the taxonomy of vehicular attack surfaces, evaluates FL topologies, and maps adversarial threats such as poisoning and inference attacks.
  arXiv  Detail & Related papers  (2026-07-12T20:40:37Z)
• COVTrack++: Learning Open-Vocabulary Multi-Object Tracking from Continuous Videos via a Synergistic Paradigm [59.26203051651017]
  C-TAO is the first continuously annotated training set for Open-Vocabulary Multi-Object Tracking (OVMOT)<n>For the framework bottleneck, we propose COVTrack++, a synergistic framework that achieves a bidirectional reciprocal mechanism between detection and association through three modules.<n>Experiments on TAO demonstrate state-of-the-art performance, with novel TETA reaching 35.4% and 30.5% on validation and test sets, improving novel AssocA by 4.8% and novel LocA by 5.8% over previous methods.
  arXiv  Detail & Related papers  (2026-03-25T07:20:27Z)
• SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine [8.662817298688147]
  RefAV is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios.<n>SMc2f is a pipeline that employs vision-language models (VLMs) for coarse image-text filtering.<n> Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.
  arXiv  Detail & Related papers  (2026-01-17T11:25:55Z)
• Federated Spatiotemporal Graph Learning for Passive Attack Detection in Smart Grids [2.721477719641864]
  This paper introduces a graph-centric, multimodal detector that fuses physical-layer and behavioral indicators over temporal windows to detect passive attacks.<n>The model achieves a testing accuracy of 98.32% per-timestep and 93.35% per-sequence at 0.15% FPR.
  arXiv  Detail & Related papers  (2025-09-29T08:52:30Z)
• LeakSealer: A Semisupervised Defense for LLMs Against Prompt Injection and Leakage Attacks [7.115093658017371]
  LeakSealer is a model-agnostic framework that combines static analysis for forensic insights with dynamic defenses in a Human-In-The-Loop pipeline.<n>We empirically evaluate LeakSealer under two scenarios: (1) jailbreak attempts, employing a public benchmark dataset, and (2) PII leakage, supported by a curated dataset of labeled LLM interactions.
  arXiv  Detail & Related papers  (2025-08-01T13:04:28Z)
• SAM2Auto: Auto Annotation Using FLASH [13.638155035372835]
  Vision-Language Models (VLMs) lag behind Large Language Models due to the scarcity of annotated datasets.<n>We introduce SAM2Auto, the first fully automated annotation pipeline for video datasets requiring no human intervention or dataset-specific training.<n>Our system employs statistical approaches to minimize detection errors while ensuring consistent object tracking throughout entire video sequences.
  arXiv  Detail & Related papers  (2025-06-09T15:15:15Z)
• Real-Time Anomaly Detection in Video Streams [0.0]
  This thesis is part of a CIFRE agreement between the company Othello and the LIASD laboratory.<n>The objective is to develop an artificial intelligence system that can detect real-time dangers in a video stream.
  arXiv  Detail & Related papers  (2024-11-29T14:24:33Z)
• Benchmarking the Robustness of LiDAR-Camera Fusion for 3D Object Detection [58.81316192862618]
  Two critical sensors for 3D perception in autonomous driving are the camera and the LiDAR. fusing these two modalities can significantly boost the performance of 3D perception models. We benchmark the state-of-the-art fusion methods for the first time.
  arXiv  Detail & Related papers  (2022-05-30T09:35:37Z)
• Unsupervised Person Re-Identification with Wireless Positioning under Weak Scene Labeling [131.18390399368997]
  We propose to explore unsupervised person re-identification with both visual data and wireless positioning trajectories under weak scene labeling. Specifically, we propose a novel unsupervised multimodal training framework (UMTF), which models the complementarity of visual data and wireless information. Our UMTF contains a multimodal data association strategy (MMDA) and a multimodal graph neural network (MMGN)
  arXiv  Detail & Related papers  (2021-10-29T08:25:44Z)
• SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving [94.11868795445798]
  We release a Large-Scale Object Detection benchmark for Autonomous driving, named as SODA10M, containing 10 million unlabeled images and 20K images labeled with 6 representative object categories. To improve diversity, the images are collected every ten seconds per frame within 32 different cities under different weather conditions, periods and location scenes. We provide extensive experiments and deep analyses of existing supervised state-of-the-art detection models, popular self-supervised and semi-supervised approaches, and some insights about how to develop future models.
  arXiv  Detail & Related papers  (2021-06-21T13:55:57Z)
• Video-based Person Re-identification without Bells and Whistles [49.51670583977911]
  Video-based person re-identification (Re-ID) aims at matching the video tracklets with cropped video frames for identifying the pedestrians under different cameras. There exists severe spatial and temporal misalignment for those cropped tracklets due to the imperfect detection and tracking results generated with obsolete methods. We present a simple re-Detect and Link (DL) module which can effectively reduce those unexpected noise through applying the deep learning-based detection and tracking on the cropped tracklets.
  arXiv  Detail & Related papers  (2021-05-22T10:17:38Z)
• SoDA: Multi-Object Tracking with Soft Data Association [75.39833486073597]
  Multi-object tracking (MOT) is a prerequisite for a safe deployment of self-driving cars. We propose a novel approach to MOT that uses attention to compute track embeddings that encode dependencies between observed objects.
  arXiv  Detail & Related papers  (2020-08-18T03:40:25Z)
• Contextual-Bandit Anomaly Detection for IoT Data in Distributed Hierarchical Edge Computing [65.78881372074983]
  IoT devices can hardly afford complex deep neural networks (DNN) models, and offloading anomaly detection tasks to the cloud incurs long delay. We propose and build a demo for an adaptive anomaly detection approach for distributed hierarchical edge computing (HEC) systems. We show that our proposed approach significantly reduces detection delay without sacrificing accuracy, as compared to offloading detection tasks to the cloud.
  arXiv  Detail & Related papers  (2020-04-15T06:13:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.