SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents

• URL: http://arxiv.org/abs/2607.14543v1
• Date: Thu, 16 Jul 2026 03:59:44 GMT
• Title: SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents
• Abstract summary: We introduce SAFERELBENCH, a spatial-relation-aware safety benchmark with 507 executable evaluation samples.<n>Using SAFERELBENCH to evaluate seven open- and closed-source VLM-driven embodied agents, we find a substantial gap between task success and process-level safety compliance.
• Score: 29.05559307748216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are increasingly used as the reasoning backbone of embodied agents, enabling robots to interpret visual scenes, follow language instructions, and plan multi-step actions. In household environments, however, safety depends not only on recognizing objects, but also on how actions change the physical scene over time. Existing embodied safety evaluations largely focus on static risk recognition, unsafe instruction refusal, or final-state task completion. As a result, process-level safety failures induced by spatial relations such as support, containment, and proximity remain insufficiently studied. To address this gap, we introduce SAFERELBENCH, a spatial-relation-aware safety benchmark with 507 executable evaluation samples, including 248 spatial-relation samples and 259 non-spatial control samples. Using SAFERELBENCH to evaluate seven open- and closed-source VLM-driven embodied agents, we find a substantial gap between task success and process-level safety compliance: models often complete the requested task while violating process-level safety constraints. Unlike prior benchmarks, SAFERELBENCH explicitly tests whether agents satisfy safety conditions before risk-prone actions, making spatial relations a core dimension in embodied safety assessment. More broadly, our results show that safe embodied intelligence requires not only stronger perception and planning, but also reliable reasoning about how object relations shape risk during interaction.

Related papers

• ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models [7.1396544907031165]
  We introduce ForesightSafety-VLA, a diagnostic benchmark that makes safety the primary evaluation target for vision-trivial-action (VLA) systems.<n>We define a 13-category safety taxonomy covering physical interaction safety (Safe-Core), instruction-side safety (Safe-Lang), and perception-side safety (Safe-Vis)<n>We instantiate 66 safety-augmented base scenarios in RoboTwin across 5 embodiments and report results on representative VLA baselines.
  arXiv  Detail & Related papers  (2026-06-25T14:19:36Z)
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces [18.650640826228862]
  SABER is a benchmark for environment-aware operational safety in large language models.<n>It places models in realistic agent-style projects and evaluates safety from the final environment state after a sequence of actions.<n>Our evaluations show that even the best-performing model has more than a 54% harmful safety-violation rate.
  arXiv  Detail & Related papers  (2026-05-31T16:06:02Z)
• SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models [46.968844120076916]
  We introduce SafetyALFRED, built upon the embodied agent benchmark ALFRED, augmented with six categories of real-world kitchen hazards.<n>We evaluate eleven state-of-the-art models on not only hazard recognition, but also active risk mitigation through embodied planning.
  arXiv  Detail & Related papers  (2026-04-21T16:27:20Z)
• HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models [87.35765363039638]
  We introduce HazardArena, a benchmark designed to evaluate semantic safety in Vision-Language-Action models.<n>We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts.<n>We propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge.
  arXiv  Detail & Related papers  (2026-04-14T08:32:02Z)
• DRAFT: Task Decoupled Latent Reasoning for Agent Safety [59.46137757545185]
  We propose DRAFT (Task Decoupled Latent Reasoning for Agent Safety), a latent reasoning framework that decouples safety judgment into two trainable stages.<n>An Extractor distills the full trajectory into a compact continuous latent draft, and a Reasoner that jointly attends to the draft and the original trajectory to predict safety.<n>DRAFT consistently outperforms strong baselines, improving accuracy from 63.27% (LoRA) to 91.18% averaged over benchmarks.
  arXiv  Detail & Related papers  (2026-02-11T07:45:14Z)
• BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments [22.32760987248309]
  We present BeSafe-Bench (BSB), a benchmark for exposing behavioral safety risks of situated agents in functional environments.<n>BSB covers four representative domains: Web, Mobile, Embodied VLM, and Embodied VLA.<n>Using functional environments, we construct a diverse instruction space by augmenting tasks with nine categories of safety-critical risks.
  arXiv  Detail & Related papers  (2026-01-30T03:41:57Z)
• RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic [56.38397499463889]
  Embodied agents powered by vision-language models (VLMs) are increasingly capable of executing complex real-world tasks.<n>However, they remain vulnerable to hazardous instructions that may trigger unsafe behaviors.<n>We propose RoboSafe, a runtime safeguard for embodied agents through executable predicate-based safety logic.
  arXiv  Detail & Related papers  (2025-12-24T15:01:26Z)
• How Brittle is Agent Safety? Rethinking Agent Risk under Intent Concealment and Task Complexity [55.441602598245744]
  Current safety evaluations for LLM-driven agents primarily focus on atomic harms, failing to address sophisticated threats where malicious intent is concealed or diluted within complex tasks.<n>We address this gap with a two-dimensional analysis of agent safety brittleness under the pressures of intent concealment and task complexity.<n>Our findings reveal two critical phenomena: safety alignment degrades sharply and predictably as intent becomes obscured, and a "Complexity Paradox" emerges, where agents seem safer on harder tasks only due to capability limitations.
  arXiv  Detail & Related papers  (2025-11-11T17:27:27Z)
• IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks [30.535665641990114]
  We present IS-Bench, the first multi-modal benchmark designed for interactive safety.<n>It features 161 challenging scenarios with 388 unique safety risks instantiated in a high-fidelity simulator.<n>It facilitates a novel process-oriented evaluation that verifies whether risk mitigation actions are performed before/after specific risk-prone steps.
  arXiv  Detail & Related papers  (2025-06-19T15:34:46Z)
• Agent-SafetyBench: Evaluating the Safety of LLM Agents [72.92604341646691]
  We introduce Agent-SafetyBench, a benchmark designed to evaluate the safety of large language models (LLMs)<n>Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions.<n>Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%.
  arXiv  Detail & Related papers  (2024-12-19T02:35:15Z)
• SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering [56.92068213969036]
  Safety alignment is indispensable for Large Language Models (LLMs) to defend threats from malicious instructions.<n>Recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue.<n>We propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns.
  arXiv  Detail & Related papers  (2024-08-21T10:01:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.