Autonomous Construction-Site Safety Inspection Using Mobile Robots: A Multilayer VLM-LLM Pipeline

• URL: http://arxiv.org/abs/2512.13974v1
• Date: Tue, 16 Dec 2025 00:25:31 GMT
• Title: Autonomous Construction-Site Safety Inspection Using Mobile Robots: A Multilayer VLM-LLM Pipeline
• Abstract summary: This paper aims to connect what a robot sees during autonomous navigation to the safety rules that are common in construction sites.<n>On the robotics side, SLAM and autonomous navigation provide repeatable coverage and targeted revisits via waypoints.<n>On AI side, a Vision Language Model (VLM)-based layer produces scene descriptions; a retrieval component powered grounds those descriptions in OSHA and site policies.<n>Another VLM-based layer assesses the safety situation based on rules; and finally Large Language Model (LLM) layer generates safety reports based on previous outputs.
• Score: 2.9360071145551068
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Construction safety inspection remains mostly manual, and automated approaches still rely on task-specific datasets that are hard to maintain in fast-changing construction environments due to frequent retraining. Meanwhile, field inspection with robots still depends on human teleoperation and manual reporting, which are labor-intensive. This paper aims to connect what a robot sees during autonomous navigation to the safety rules that are common in construction sites, automatically generating a safety inspection report. To this end, we proposed a multi-layer framework with two main modules: robotics and AI. On the robotics side, SLAM and autonomous navigation provide repeatable coverage and targeted revisits via waypoints. On AI side, a Vision Language Model (VLM)-based layer produces scene descriptions; a retrieval component powered grounds those descriptions in OSHA and site policies; Another VLM-based layer assesses the safety situation based on rules; and finally Large Language Model (LLM) layer generates safety reports based on previous outputs. The framework is validated with a proof-of-concept implementation and evaluated in a lab environment that simulates common hazards across three scenarios. Results show high recall with competitive precision compared to state-of-the-art closed-source models. This paper contributes a transparent, generalizable pipeline that moves beyond black-box models by exposing intermediate artifacts from each layer and keeping the human in the loop. This work provides a foundation for future extensions to additional tasks and settings within and beyond construction context.

Related papers

• Harness Robotic OS: A Unified Embodied-Agent Runtime for Closed-Loop Quadruped Inspection [4.737450468943755]
  This paper presents textitHarness Robotic OS (HROS), a unified embodied-agent runtime, and Argos, its realization for residential-community inspection.<n>HROS organizes the system into robot runtime, embodied autonomy skills, cognitive agent, and interaction and operations planes.<n>Experiments in a residential property environment achieved 100% waypoint reachability, outdoor localization error below 10cm, local obstacle-response latency below 200ms, and 99% success in alarm delivery.
  arXiv  Detail & Related papers  (2026-09-10T08:25:46Z)
• Verifiable Foundation Models for Robot Safety [9.839166424557009]
  We present FEARL (Foundation-Enabled Assured Robot Learning), a framework that addresses this tension through a modular architectural decomposition.<n>FEARL separates the policy into a large Controller (C) responsible for high-dimensional perception and task reasoning, and a small Safety module (S) that receives low-dimensional observations from dedicated safety sensors.<n>This makes formal analysis tractable with existing tools while preserving the Controller's expressive power for task reasoning.
  arXiv  Detail & Related papers  (2026-06-22T03:10:51Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• The Unified Autonomy Stack: Toward a Blueprint for Generalizable Robot Autonomy [13.105188128582567]
  We introduce and open-source the Unified Autonomy Stack, a system-level solution that enables resilient autonomy across aerial and ground robot morphologies.<n>The architecture centers on three modules -- multi-modal perception, multi-behavior planning, and multi-layered safe navigation.<n>The stack has been field-tested and validated on both aerial (rotorcraft) and ground (legged) robots.
  arXiv  Detail & Related papers  (2026-05-12T20:39:35Z)
• AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act [27.922630781100864]
  We present AgenticLab, a model-agnostic robot agent platform and benchmark for open-world manipulation.<n>We benchmark state-of-the-art VLM-based agents on real-robot tasks in unstructured environments.<n>Our benchmark reveals several failure modes that offline vision-language tests fail to capture.
  arXiv  Detail & Related papers  (2026-02-02T05:30:14Z)
• SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine [8.662817298688147]
  RefAV is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios.<n>SMc2f is a pipeline that employs vision-language models (VLMs) for coarse image-text filtering.<n> Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.
  arXiv  Detail & Related papers  (2026-01-17T11:25:55Z)
• OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows [77.95511352806261]
  Computer-using agents powered by Vision-Language Models (VLMs) have demonstrated human-like capabilities in operating digital environments like mobile platforms.<n>We propose OS-Sentinel, a novel hybrid safety detection framework that combines a Formal Verifier for detecting explicit system-level violations with a Contextual Judge for assessing contextual risks and agent actions.
  arXiv  Detail & Related papers  (2025-10-28T13:22:39Z)
• Constrained Decoding for Robotics Foundation Models [12.916330118607918]
  We introduce SafeDec, a constrained decoding framework for autoregressive robot foundation models.<n>Task-specific safety rules are expressed as Signal Temporal Logic (STL) formulas and are enforced at inference time with minimal overhead.<n>Our method ensures that generated actions provably satisfy STL specifications under assumed dynamics at runtime without retraining.
  arXiv  Detail & Related papers  (2025-09-01T19:17:40Z)
• OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use [101.57043903478257]
  The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations.<n>With the evolution of (multi-modal) large language models ((M)LLMs), this dream is closer to reality.<n>This survey aims to consolidate the state of OS Agents research, providing insights to guide both academic inquiry and industrial development.
  arXiv  Detail & Related papers  (2025-08-06T14:33:45Z)
• General-Purpose Robotic Navigation via LVLM-Orchestrated Perception, Reasoning, and Acting [5.291702442384798]
  Agentic Robotic Navigation Architecture (ARNA) is a general-purpose framework that equips an LVLM-based agent with a library of perception, reasoning, and navigation tools.<n>At runtime, the agent autonomously defines and executes task-specific navigation tools drawn from modern robotic stacks.<n>ARNA outperforms state-of-the-art EQA-specific approaches.
  arXiv  Detail & Related papers  (2025-06-20T20:06:14Z)
• AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructions [64.85086226439954]
  We present SAFE, a benchmark for assessing the safety of embodied VLM agents on hazardous instructions.<n> SAFE comprises three components: SAFE-THOR, SAFE-VERSE, and SAFE-DIAGNOSE.<n>We uncover systematic failures in translating hazard recognition into safe planning and execution.
  arXiv  Detail & Related papers  (2025-06-17T16:37:35Z)
• FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks [52.47895046206854]
  FieldWorkArena is a benchmark for agentic AI targeting real-world field work.<n>This paper defines a new action space that agentic AI should possess for real world work environment benchmarks.
  arXiv  Detail & Related papers  (2025-05-26T08:21:46Z)
• SafeAuto: Knowledge-Enhanced Safe Autonomous Driving with Multimodal Foundation Models [63.71984266104757]
  We propose SafeAuto, a framework that enhances MLLM-based autonomous driving by incorporating both unstructured and structured knowledge.<n>To explicitly integrate safety knowledge, we develop a reasoning component that translates traffic rules into first-order logic.<n>Our Multimodal Retrieval-Augmented Generation model leverages video, control signals, and environmental attributes to learn from past driving experiences.
  arXiv  Detail & Related papers  (2025-02-28T21:53:47Z)
• Interactive Planning Using Large Language Models for Partially Observable Robotics Tasks [54.60571399091711]
  Large Language Models (LLMs) have achieved impressive results in creating robotic agents for performing open vocabulary tasks. We present an interactive planning technique for partially observable tasks using LLMs.
  arXiv  Detail & Related papers  (2023-12-11T22:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.