SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation

• URL: http://arxiv.org/abs/2605.12386v1
• Date: Tue, 12 May 2026 16:49:28 GMT
• Title: SafeManip: A Property-Driven Benchmark for Temporal Safety Evaluation in Robotic Manipulation
• Abstract summary: We introduce SafeManip, a property-driven benchmark to explicitly evaluate temporal safety properties in robotic manipulation.<n>SafeManip defines reusable safety templates over finite executions using Linear Temporal Logic over finite traces (LTLf)<n>We evaluate SafeManip on six vision-language-action policies, including $_0.5$, GR00T, and their training variants, across 50 RoboCasa365 household tasks.
• Score: 30.582295885898997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robotic manipulation is typically evaluated by task success, but successful completion does not guarantee safe execution. Many safety failures are temporal: a robot may touch a clean surface after contamination or release an object before it is fully inside an enclosure. We introduce SafeManip, a property-driven benchmark to explicitly evaluate temporal safety properties in robotic manipulation, moving beyond prior evaluations that largely focus on task completion or per-state constraint violations. SafeManip defines reusable safety templates over finite executions using Linear Temporal Logic over finite traces (LTLf). It maps observed rollouts to symbolic predicate traces and evaluates them with LTLf-based monitors. Its property suite covers eight manipulation safety categories: collision and contact safety, grasp stability, release stability, cross-contamination, action onset, mechanism recovery, object containment, and enclosure access. Templates can be instantiated with task-specific objects, fixtures, regions, or skills, allowing the same safety specifications to generalize across tasks and environments. We evaluate SafeManip on six vision-language-action policies, including $π_0$, $π_{0.5}$, GR00T, and their training variants, across 50 RoboCasa365 household tasks. Results show that even strong models often behave unsafely. Task-success gains do not reliably translate into safer execution: many successful rollouts remain unsafe, while longer-horizon or more complex tasks expose more violations. SafeManip provides a reusable evaluation layer for diagnosing temporal safety failures and measuring safe success beyond task completion.

Related papers

• SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects [56.16187327142104]
  We present SoftVTBench, a safety-aware visuo-tactile benchmark for deformable object manipulation.<n>Built in Isaac Sim with finite-element-simulated deformable objects, SoftVTBench provides multi-view RGB observations, tactile sensing with marker motion, proprioception, and language instructions.
  arXiv  Detail & Related papers  (2026-07-05T11:12:29Z)
• Self-Evolving Just-In-Time Memory for Proactive Embodied Safety [13.342067741014455]
  Existing safety approaches rely on runtime guardrails to block unsafe actions or induce excessive caution.<n>We introduce the Self-Evolving Just-In-Time Memory framework, which reframes embodied safety from progress-stalling guardrails to proactive hazard mitigation.
  arXiv  Detail & Related papers  (2026-06-26T14:26:24Z)
• OSGuard: A Benchmark for Safety in Computer-Use Agents [12.395093977641581]
  OSGuard is a benchmark suite for evaluating safety in computer-use agents under benign, unchanged user instructions.<n> OSGuard contains an action-level benchmark for local guardrail decisions and a risk-augmented execution suite for end-to-end evaluation.
  arXiv  Detail & Related papers  (2026-06-13T00:32:24Z)
• VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring [60.53395558502203]
  Vision-Language Embodied Safety Agent (VLESA) monitors human activities from egocentric video.<n>VLESA addresses intent-dependent safety where identical actions can be safe or dangerous depending on context.<n>An intent-action prediction agent is proposed to jointly infer goals and predict future actions from video.
  arXiv  Detail & Related papers  (2026-06-02T17:42:17Z)
• SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents [87.26967184869198]
  SeClaw is a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents.<n> benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors.
  arXiv  Detail & Related papers  (2026-06-01T14:23:42Z)
• SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models [5.9288463407781995]
  Vision-language-action (VLA) benchmarks measure whether a policy completes a requested manipulation task.<n>We present SafeVLA-Bench, a post-hoc safety-evaluation framework for existing simulator-based VLA benchmarks.
  arXiv  Detail & Related papers  (2026-05-30T15:31:57Z)
• SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces [28.615215165815297]
  We present SkillSafetyBench, a benchmark for evaluating skill-mediated safety failures.<n>We show that localized non-user attacks can consistently induce unsafe behavior.<n>Our findings suggest that agent safety depends not only on model-level alignment, but also on how agents interpret skills.
  arXiv  Detail & Related papers  (2026-05-12T12:03:54Z)
• Pre-Execution Safety Gate & Task Safety Contracts for LLM-Controlled Robot Systems [14.419994656981002]
  We introduce SafeGate, a neurosymbolic safety architecture that prevents unsafe natural language task commands from reaching robot execution.<n>In addition, we introduce Task Safety Contracts, which decomposes commands that pass through the gate into invariants, guards, and abort conditions.<n>Results show that SafeGate significantly reduces the acceptance of defective commands while maintaining a high acceptance of benign tasks.
  arXiv  Detail & Related papers  (2026-04-07T04:46:06Z)
• Internal Safety Collapse in Frontier Large Language Models [65.00730294617382]
  This work identifies a critical failure mode in frontier large language models (LLMs)<n>Under certain task conditions, models enter a state in which they continuously generate harmful content while executing otherwise benign tasks.<n>We introduce TVD, a framework that triggers ISC through domain tasks where generating harmful content is the only valid completion.
  arXiv  Detail & Related papers  (2026-03-04T12:55:34Z)
• RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic [56.38397499463889]
  Embodied agents powered by vision-language models (VLMs) are increasingly capable of executing complex real-world tasks.<n>However, they remain vulnerable to hazardous instructions that may trigger unsafe behaviors.<n>We propose RoboSafe, a runtime safeguard for embodied agents through executable predicate-based safety logic.
  arXiv  Detail & Related papers  (2025-12-24T15:01:26Z)
• Pro2Guard: Proactive Runtime Enforcement of LLM Agent Safety via Probabilistic Model Checking [8.970702398918924]
  Large Language Model (LLM) agents exhibit powerful autonomous capabilities across domains such as robotics, virtual assistants, and web automation.<n>Existing rule-based enforcement systems, such as AgentSpec, focus on developing reactive safety rules.<n>We propose Pro2Guard, a proactive runtime enforcement framework grounded in probabilistic reachability analysis.
  arXiv  Detail & Related papers  (2025-08-01T10:24:47Z)
• SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents [58.65256663334316]
  We present SafeAgentBench -- the first benchmark for safety-aware task planning of embodied LLM agents in interactive simulation environments.<n>SafeAgentBench includes: (1) an executable, diverse, and high-quality dataset of 750 tasks, rigorously curated to cover 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 9 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives.
  arXiv  Detail & Related papers  (2024-12-17T18:55:58Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.