Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification

• URL: http://arxiv.org/abs/2604.05210v1
• Date: Mon, 06 Apr 2026 22:10:05 GMT
• Title: Integration of Object Detection and Small VLMs for Construction Safety Hazard Identification
• Abstract summary: Small vision-language models (sVLMs) with fewer than 4 billion parameters offer improved efficiency but often suffer from reduced accuracy and hallucination.<n>This study proposes a detection-guided sVLM framework that integrates object detection with multimodal reasoning for contextual hazard identification.
• Score: 1.1070461196091774
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate and timely identification of construction hazards around workers is essential for preventing workplace accidents. While large vision-language models (VLMs) demonstrate strong contextual reasoning capabilities, their high computational requirements limit their applicability in near real-time construction hazard detection. In contrast, small vision-language models (sVLMs) with fewer than 4 billion parameters offer improved efficiency but often suffer from reduced accuracy and hallucination when analyzing complex construction scenes. To address this trade-off, this study proposes a detection-guided sVLM framework that integrates object detection with multimodal reasoning for contextual hazard identification. The framework first employs a YOLOv11n detector to localize workers and construction machinery within the scene. The detected entities are then embedded into structured prompts to guide the reasoning process of sVLMs, enabling spatially grounded hazard assessment. Within this framework, six sVLMs (Gemma-3 4B, Qwen-3-VL 2B/4B, InternVL-3 1B/2B, and SmolVLM-2B) were evaluated in zero-shot settings on a curated dataset of construction site images with hazard annotations and explanatory rationales. The proposed approach consistently improved hazard detection performance across all models. The best-performing model, Gemma-3 4B, achieved an F1-score of 50.6%, compared to 34.5% in the baseline configuration. Explanation quality also improved significantly, with BERTScore F1 increasing from 0.61 to 0.82. Despite incorporating object detection, the framework introduces minimal overhead, adding only 2.5 ms per image during inference. These results demonstrate that integrating lightweight object detection with small VLM reasoning provides an effective and efficient solution for context-aware construction safety hazard detection.

Related papers

• Assessing the Benefits of Combining Advanced Deep Learning Techniques for Post-Disaster Building Damage Assessment from UAV Imagery [3.016488960368601]
  We propose a hybrid framework that decouples detection from damage assessment.<n>A CV model first detects buildings and generates bounding boxes on the image that are then passed to an LVLM for damage classification and contextual interpretation.<n>We evaluate our framework on two real-world benchmarks: RescueNet and FloodNet.
  arXiv  Detail & Related papers  (2026-08-03T08:39:59Z)
• HUGE-Bench: A Benchmark for High-Level UAV Vision-Language-Action Tasks [73.62053624331227]
  Huge-Bench is a benchmark for High-Level UAV Vision-Language-Action tasks.<n>It comprises 4 real-world digital twin scenes, 8 high-level tasks, and 2.56M meters of trajectories.<n>We introduce process-oriented and collision-aware metrics to assess process fidelity, terminal accuracy, and safety.
  arXiv  Detail & Related papers  (2026-03-20T10:08:42Z)
• VIRO: Robust and Efficient Neuro-Symbolic Reasoning with Verification for Referring Expression Comprehension [51.76841625486355]
  Referring Expression (REC) aims to localize the image region corresponding to a natural-language query.<n>Recent neuro-symbolic REC approaches leverage large language models (LLMs) and vision-language models (VLMs) to perform compositional reasoning.<n>We introduce VIRO, a neuro-symbolic framework that embeds lightweight operator-level verifiers within reasoning steps.
  arXiv  Detail & Related papers  (2026-01-19T07:21:19Z)
• Noise-Robust Tiny Object Localization with Flows [63.60972031108944]
  We propose a noise-robust localization framework leveraging normalizing flows for flexible error modeling and uncertainty-guided optimization.<n>Our method captures complex, non-Gaussian prediction distributions through flow-based error modeling, enabling robust learning under noisy supervision.<n>An uncertainty-aware gradient modulation mechanism further suppresses learning from high-uncertainty, noise-prone samples, mitigating overfitting while stabilizing training.
  arXiv  Detail & Related papers  (2026-01-02T09:16:55Z)
• Automated Hazard Detection in Construction Sites Using Large Language and Vision-Language Models [0.0]
  This thesis explores a multimodal AI framework for enhancing construction safety through the combined analysis of textual and visual data.<n>Two case studies were consucted to evaluate the capabilities of large language models (LLMs) and vision-language models (VLMs) for automated hazard identification.
  arXiv  Detail & Related papers  (2025-11-13T02:23:45Z)
• Investigating Traffic Accident Detection Using Multimodal Large Language Models [3.4123736336071864]
  This research investigates the zero-shot capabilities of multimodal large language models (MLLMs) for detecting and describing traffic accidents.<n>Results show Pixtral as the top performer with an F1-score of 71% and 83% recall.<n>These findings demonstrate the substantial potential of integrating MLLMs with advanced visual analytics techniques.
  arXiv  Detail & Related papers  (2025-09-23T14:47:33Z)
• DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models [60.713908578319256]
  We propose Direct Discrepancy Learning (DDL) to optimize the detector with task-oriented knowledge.<n>Built upon this, we introduce DetectAnyLLM, a unified detection framework that achieves state-of-the-art MGTD performance.<n>MIRAGE samples human-written texts from 10 corpora across 5 text-domains, which are then re-generated or revised using 17 cutting-edge LLMs.
  arXiv  Detail & Related papers  (2025-09-15T10:59:57Z)
• Prompt to Protection: A Comparative Study of Multimodal LLMs in Construction Hazard Recognition [0.0]
  This study conducts a comparative evaluation of five state-of-the-art large language models (LLMs)<n>Each model was tested under three prompting strategies: zero-shot, few-shot, and chain-of-thought (CoT)<n>Results reveal that CoT prompting significantly influenced performance, with CoT prompting consistently producing higher accuracy across models.
  arXiv  Detail & Related papers  (2025-06-09T05:22:35Z)
• Safe-Construct: Redefining Construction Safety Violation Recognition as 3D Multi-View Engagement Task [2.0811729303868005]
  We introduce Safe-Construct, a framework that reformulates violation recognition as a 3D multi-view engagement task.<n>Safe-Construct achieves a 7.6% improvement over state-of-the-art methods across four violation types.
  arXiv  Detail & Related papers  (2025-04-15T05:21:09Z)
• Uncertainty Estimation for 3D Object Detection via Evidential Learning [63.61283174146648]
  We introduce a framework for quantifying uncertainty in 3D object detection by leveraging an evidential learning loss on Bird's Eye View representations in the 3D detector. We demonstrate both the efficacy and importance of these uncertainty estimates on identifying out-of-distribution scenes, poorly localized objects, and missing (false negative) detections.
  arXiv  Detail & Related papers  (2024-10-31T13:13:32Z)
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [76.2621758731288]
  We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation. Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA) We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
  arXiv  Detail & Related papers  (2021-08-03T17:09:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.