EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

• URL: http://arxiv.org/abs/2607.00218v1
• Date: Tue, 30 Jun 2026 21:50:49 GMT
• Title: EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards
• Abstract summary: Vision-language models (VLMs) are proposed as runtime safety guards for embodied agents in homes and factories.<n>We introduce EgoSafetyBench, an egocentric video benchmark of 1,200 robot-view scenarios annotated at half-second granularity.<n>We find that while guards reliably recognize videos containing hazards, they often miss specific hazardous moments.
• Score: 4.6400797556929545
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genuinely unsafe situations while avoiding unnecessary intervention on routine but superficially alarming activity, a distinction that binary safety benchmarks obscure. We introduce EgoSafetyBench, an egocentric video benchmark of 1,200 robot-view scenarios annotated at half-second granularity, to evaluate VLMs as streaming guards across two tracks. The situational track (800 scenarios) spans four families, from routine and safe-but-suspicious scenes to obvious and contextual hazards. The visual-channel track (400 scenarios) targets in-scene text-a sign, sticker, or label visible in the scene-that can misrepresent the physical situation, pairing each misleading sign with a truthful version to test both whether a guard flags the text as misleading and whether the text corrupts its physical-safety judgment. Both tracks use contrastive ladders: near-identical scenarios differing only in a single visible deciding cue, so a correct call must hinge on that cue rather than the overall scene type. We evaluate ten open- and closed-source VLMs. We find that while guards reliably recognize videos containing hazards, they often miss specific hazardous moments, particularly contextual hazards. Furthermore, misleading in-scene signs degrade all tested guards: vulnerable models miss up to a third of hazards, while robust models over-intervene on safe content. Matched controls reveal that apparent safety robustness often reflects indiscriminate alarming rather than true physical reasoning.

Related papers

• When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs [54.95635713318535]
  Vision-language models (VLMs) are increasingly deployed in high-stakes settings, where a response that is reasonable in general may still be unsafe for a particular user.<n>We study this problem of personalized safety in multimodal systems and introduce MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images.<n>We find that they almost always respond directly (86-99%) rather than seek missing context, and none exceeds 2.6/5 on personalized safety.<n>Motivated by this mechanism, we propose PRISM, a lightweight input monitor that uses bidirectional cross-modal modulation to predict when a query is likely to require
  arXiv  Detail & Related papers  (2026-09-03T03:58:56Z)
• GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AI [33.22835977135006]
  In embodied AI, safety risk can be latent: a benign instruction and a safe scene become hazardous only when composed.<n>We introduce GuardianBench, an instruction-contrastive benchmark grounded in international safety standards.<n> Benchmarking state-of-the-art vision-language models (VLMs) reveals instruction-insensitive verdicts.
  arXiv  Detail & Related papers  (2026-08-22T11:40:27Z)
• From Sports to Safety: Benchmarking Proactive Risk Inference in MLLMs [11.582231928519754]
  SPRINT (Sports Proactive Risk INference Testbed) is a benchmark of 2,888 real-world sports videos (2,440 accident, 448 safe controls) spanning 14 sports and 3 environmental settings.<n> Accident videos feature fine-grained annotations of early hazard cues, accident timing, and hierarchical causes; safe videos are manually verified as accident-free and serve to diagnose prompt-induced false alarms.
  arXiv  Detail & Related papers  (2026-08-06T03:25:39Z)
• Hazard or Anomaly? Evaluating VLMs for Understanding Dangers and Discrepancies [3.227001704699474]
  Vision-Language Models (VLMs) can interpret complex scenes and communicate safety-relevant information.<n>Current evaluations often frame danger recognition as a binary decision (Safe/Unsafe)<n>We introduce an explicit distinction between hazard and anomaly, and by separately recognizing hazardous and anomalous states.
  arXiv  Detail & Related papers  (2026-07-18T02:24:06Z)
• PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning [54.70033525672316]
  Video-capable large language models (MLLMs) could serve as always-on safety monitors that issue warnings during this window.<n>We present PaSBench-Video, a 740-video benchmark with 481 risk and 259 no-risk videos across four domains.<n>We find that no model exceeds 20.0% on our strictest metric, and recall is tightly coupled with false-positive rate.
  arXiv  Detail & Related papers  (2026-06-01T16:14:01Z)
• Discovering Agentic Safety Specifications from 1-Bit Danger Signals [6.599344783327054]
  EPO-Safe is a framework where an agent iteratively generates action plans, receives sparse binary danger warnings, and evolves a natural language behavioral specification through reflection.<n>We show that EPO-Safe can perform safety reasoning from a strictly impoverished signal in structured, low-dimensional environments.<n>We show that standard reward-driven reflection actively degrades safety, proving that reflection must be paired with a dedicated safety channel.
  arXiv  Detail & Related papers  (2026-04-25T08:35:36Z)
• HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models [87.35765363039638]
  We introduce HazardArena, a benchmark designed to evaluate semantic safety in Vision-Language-Action models.<n>We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts.<n>We propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge.
  arXiv  Detail & Related papers  (2026-04-14T08:32:02Z)
• SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues [50.928203120795786]
  We study whether multimodal safety behavior in vision-language models can be steered by simple semantic cues.<n>We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content.
  arXiv  Detail & Related papers  (2026-03-19T16:18:00Z)
• RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic [56.38397499463889]
  Embodied agents powered by vision-language models (VLMs) are increasingly capable of executing complex real-world tasks.<n>However, they remain vulnerable to hazardous instructions that may trigger unsafe behaviors.<n>We propose RoboSafe, a runtime safeguard for embodied agents through executable predicate-based safety logic.
  arXiv  Detail & Related papers  (2025-12-24T15:01:26Z)
• HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model [58.12612140992874]
  We introduce a holistic safety dataset and benchmark, textbfHoliSafe, that spans all five safe/unsafe image-text combinations.<n>We also propose a novel modular framework for enhancing VLM safety with a visual guard module (VGM) designed to assess the harmfulness of input images.<n> Experiments show that Safe-VLM with VGM, trained on our HoliSafe, achieves state-of-the-art safety performance across multiple VLM benchmarks.
  arXiv  Detail & Related papers  (2025-06-05T07:26:34Z)
• Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognition [12.054081112688074]
  Vision-Language Models (VLMs) have shown capabilities in interpreting visual content, but their reliability in safety-critical scenarios remains insufficiently explored.<n>We introduce VERI, a diagnostic benchmark comprising 200 synthetic images (100 contrastive pairs) and an additional 50 real-world images (25 pairs) for validation.<n>Each emergency scene is paired with a visually similar but safe counterpart through human verification.
  arXiv  Detail & Related papers  (2025-05-21T10:57:40Z)
• Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs [45.265397990158846]
  Video-SafetyBench is the first benchmark designed to evaluate the safety of LVLMs under video-text attacks.<n>It comprises 2,264 video-text pairs spanning 48 fine-grained unsafe categories.<n>To generate semantically accurate videos for safety evaluation, we design a controllable pipeline that decomposes video semantics into subject images and motion text.
  arXiv  Detail & Related papers  (2025-05-17T05:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.