SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models

• URL: http://arxiv.org/abs/2606.00773v1
• Date: Sat, 30 May 2026 15:31:57 GMT
• Title: SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
• Abstract summary: Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task.<n>We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks.
• Score: 5.9288463407781995
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task, but binary success can hide safety-relevant trajectory behavior: reaching the goal while applying excessive contact, disturbing bystander objects, destabilizing the held object, or entering robot self-contact. We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks. It formalizes task-aware safety requirements as Signal Temporal Logic (STL) specifications and reports native success with two unsafe-success metrics: Succ-But-Unsafe (SBU), the fraction of rollouts that both succeed and violate safety, and Violation Severity Index (VSI), a bounded worst-violation depth score. We instantiate SafeVLA-Bench on LIBERO and RoboCasa-365, evaluating nine policy-benchmark entries across tabletop and kitchen manipulation tasks. High task success does not imply safe execution: high-SR tabletop baselines still leave 13 to 15 percent unsafe-episode rates,and 36 to 56 percent of successful RoboCasa-365 rollouts violate at least one active safety clause. Project page: https://safevla.org.

Related papers

• SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents [29.05559307748216]
  We introduce SAFERELBENCH, a spatial-relation-aware safety benchmark with 507 executable evaluation samples.<n>Using SAFERELBENCH to evaluate seven open- and closed-source VLM-driven embodied agents, we find a substantial gap between task success and process-level safety compliance.
  arXiv  Detail & Related papers  (2026-07-16T03:59:44Z)
• SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects [56.16187327142104]
  We present SoftVTBench, a safety-aware visuo-tactile benchmark for deformable object manipulation.<n>Built in Isaac Sim with finite-element-simulated deformable objects, SoftVTBench provides multi-view RGB observations, tactile sensing with marker motion, proprioception, and language instructions.
  arXiv  Detail & Related papers  (2026-07-05T11:12:29Z)
• ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models [7.1396544907031165]
  We introduce ForesightSafety-VLA, a diagnostic benchmark that makes safety the primary evaluation target for vision-trivial-action (VLA) systems.<n>We define a 13-category safety taxonomy covering physical interaction safety (Safe-Core), instruction-side safety (Safe-Lang), and perception-side safety (Safe-Vis)<n>We instantiate 66 safety-augmented base scenarios in RoboTwin across 5 embodiments and report results on representative VLA baselines.
  arXiv  Detail & Related papers  (2026-06-25T14:19:36Z)
• VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring [60.53395558502203]
  Vision-Language Embodied Safety Agent (VLESA) monitors human activities from egocentric video.<n>VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context.<n>An intent-action prediction agent is proposed to jointly infer goals and predict future actions from video.
  arXiv  Detail & Related papers  (2026-06-02T17:42:17Z)
• SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents [87.26967184869198]
  SeClaw is a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents.<n> benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors.
  arXiv  Detail & Related papers  (2026-06-01T14:23:42Z)
• SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation [30.582295885898997]
  We introduce SafeManip, a property-driven benchmark to explicitly evaluate temporal safety properties in robotic manipulation.<n>SafeManip defines reusable safety templates over finite executions using Linear Temporal Logic over finite traces (LTLf)<n>We evaluate SafeManip on six vision-language-action policies, including $_0.5$, GR00T, and their training variants, across 50 RoboCasa365 household tasks.
  arXiv  Detail & Related papers  (2026-05-12T16:49:28Z)
• Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMs [61.01470415470677]
  Large vision-language models (LVLMs) exhibit remarkable capabilities in cross-modal tasks but face significant safety challenges.<n>Existing benchmarks are hindered by their labor-intensive construction process, static complexity, and limited discriminative power.<n>We propose VLSafetyBencher, the first automated system for LVLM safety benchmarking.
  arXiv  Detail & Related papers  (2026-01-27T11:51:30Z)
• RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic [56.38397499463889]
  Embodied agents powered by vision-language models (VLMs) are increasingly capable of executing complex real-world tasks.<n>However, they remain vulnerable to hazardous instructions that may trigger unsafe behaviors.<n>We propose RoboSafe, a runtime safeguard for embodied agents through executable predicate-based safety logic.
  arXiv  Detail & Related papers  (2025-12-24T15:01:26Z)
• SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models [66.71948519280669]
  Multimodal Large Reasoning Models (MLRMs) demonstrate impressive crossmodal reasoning but often amplify safety risks under adversarial prompts.<n> Existing defenses mainly act at the output level and do not constrain the reasoning process, leaving models to implicit risks.<n>We propose SaFeR-VLM, which integrates four components and supports dynamic and interpretable safety decisions beyond surface-level filtering.
  arXiv  Detail & Related papers  (2025-10-08T10:39:12Z)
• SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents [7.975014390527644]
  Embodied agents powered by large language models (LLMs) inherit advanced planning capabilities; however, their direct interaction with the physical world exposes them to safety vulnerabilities.<n>We present SafeMindBench, a multimodal benchmark with 5,558 samples spanning four task categories (Instr-Risk, Env-Risk, Order-Fix, Req-Align) across high-risk scenarios such as sabotage, harm, privacy, and illegal behavior.<n>We introduce SafeMindAgent, a modular Planner-Executor architecture integrated with three cascaded safety modules, which incorporate safety constraints into the reasoning process.
  arXiv  Detail & Related papers  (2025-09-30T07:24:04Z)
• SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning [30.037309138373754]
  Vision-language-action models (VLAs) show potential as generalist robot policies.<n>These models pose extreme safety challenges during real-world deployment, including the risk of harm to the environment, the robot itself, and humans.<n>We address this by exploring an integrated safety approach (ISA), systematically modeling safety requirements, then actively eliciting diverse unsafe behaviors.
  arXiv  Detail & Related papers  (2025-03-05T13:16:55Z)
• SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents [58.65256663334316]
  We present SafeAgentBench -- the first benchmark for safety-aware task planning of embodied LLM agents in interactive simulation environments.<n>SafeAgentBench includes: (1) an executable, diverse, and high-quality dataset of 750 tasks, rigorously curated to cover 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 9 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives.
  arXiv  Detail & Related papers  (2024-12-17T18:55:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.