EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models

• URL: http://arxiv.org/abs/2608.23313v1
• Date: Mon, 24 Aug 2026 14:32:25 GMT
• Title: EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models
• Abstract summary: We introduce EviSafe, an evidence-grounded framework for VLM safety.<n>It jointly evaluates natural user-facing behavior, explicit grounding in textual and visual evidence, and behavioral sensitivity to counterfactual changes.<n>EviSafeBench instantiates the framework as a controlled benchmark with 1,181 gold image-text scenarios.
• Score: 3.659020283346239
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language model safety benchmarks typically evaluate only final responses: whether a model refuses, warns, or complies. This outcome-level view cannot tell whether a model is safe for the right multimodal reason. Safelooking behavior may reflect keyword-triggered refusal, missed visual hazards, or over-refusal of benign-sensitive inputs. We introduce EviSafe, an evidence-grounded framework for VLM safety that jointly evaluates natural user-facing behavior, explicit grounding in textual and visual evidence, and behavioral sensitivity to counterfactual changes in safety-critical evidence. EviSafeBench instantiates the framework as a controlled benchmark with 1,181 gold image-text scenarios and 2,452 targeted counterfactual variants across eight safety domains and eight risk-source types. Each scenario includes a gold safety decision, evidence annotations, a safe-response policy, and counterfactual interventions. The three-probe protocol queries models with natural-response, evidencereporting, and counterfactual-response prompts, then scores them using an evidence-aware judge. Across eleven evaluated VLMs, natural severity accuracy ranges from 27.6% to 52.8%, relaxed diagnostic consistency from 6.1% to 29.3%, and unsafe-to-safe counterfactual transition success from 30.4% to 58.4%. These gaps show that the evaluated VLMs are not reliably safe for the right multimodal reason and motivate evaluation beyond refusal counts.

Related papers

• SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation [0.4779196219827507]
  We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures.<n>Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models.<n>Results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.
  arXiv  Detail & Related papers  (2026-08-17T04:25:58Z)
• ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models [7.1396544907031165]
  We introduce ForesightSafety-VLA, a diagnostic benchmark that makes safety the primary evaluation target for vision-trivial-action (VLA) systems.<n>We define a 13-category safety taxonomy covering physical interaction safety (Safe-Core), instruction-side safety (Safe-Lang), and perception-side safety (Safe-Vis)<n>We instantiate 66 safety-augmented base scenarios in RoboTwin across 5 embodiments and report results on representative VLA baselines.
  arXiv  Detail & Related papers  (2026-06-25T14:19:36Z)
• Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering [18.59581013500335]
  Large reasoning models expose chain-of-thought-like reasoning for transparency, verification, and deliberate problem solving.<n>This creates a safety blind spot: harmful or policy-violating content may appear in reasoning traces even when final answers appear safe.<n>We test whether final-answer safety is a sufficient proxy for the full reasoning-answer trajectory by scoring both stages under a unified twenty-principle safety rubric.
  arXiv  Detail & Related papers  (2026-05-07T05:12:56Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• ProGuard: Towards Proactive Multimodal Safeguard [48.89789547707647]
  ProGuard is a vision-language proactive guard that identifies and describes out-of-distribution (OOD) safety risks.<n>We first construct a modality-balanced dataset of 87K samples, each annotated with both binary safety labels and risk categories.<n>We then train our vision-language base model purely through reinforcement learning to achieve efficient and concise reasoning.
  arXiv  Detail & Related papers  (2025-12-29T16:13:23Z)
• Think-Reflect-Revise: A Policy-Guided Reflective Framework for Safety Alignment in Large Vision Language Models [58.17589701432514]
  Think-Reflect-Revise (TRR) is a training framework designed to enhance the safety alignment of Large Vision Language Models (LVLMs)<n>We first build a Reflective Safety Reasoning (ReSafe) dataset with 5,000 examples that follow a think-reflect-revise process.<n>We then fine-tune the target model using the ReSafe dataset to initialize reflective behavior, and finally reinforce policy-guided reflection through reinforcement learning.
  arXiv  Detail & Related papers  (2025-12-08T03:46:03Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models [66.71948519280669]
  Multimodal Large Reasoning Models (MLRMs) demonstrate impressive crossmodal reasoning but often amplify safety risks under adversarial prompts.<n> Existing defenses mainly act at the output level and do not constrain the reasoning process, leaving models to implicit risks.<n>We propose SaFeR-VLM, which integrates four components and supports dynamic and interpretable safety decisions beyond surface-level filtering.
  arXiv  Detail & Related papers  (2025-10-08T10:39:12Z)
• Better Safe Than Sorry? Overreaction Problem of Vision Language Models in Visual Emergency Recognition [12.054081112688074]
  Vision-Language Models (VLMs) have shown capabilities in interpreting visual content, but their reliability in safety-critical scenarios remains insufficiently explored.<n>We introduce VERI, a diagnostic benchmark comprising 200 synthetic images (100 contrastive pairs) and an additional 50 real-world images (25 pairs) for validation.<n>Each emergency scene is paired with a visually similar but safe counterpart through human verification.
  arXiv  Detail & Related papers  (2025-05-21T10:57:40Z)
• Can't See the Forest for the Trees: Benchmarking Multimodal Safety Awareness for Multimodal LLMs [56.440345471966666]
  Multimodal Large Language Models (MLLMs) have expanded the capabilities of traditional language models by enabling interaction through both text and images.<n>This paper introduces MMSafeAware, the first comprehensive multimodal safety awareness benchmark designed to evaluate MLLMs across 29 safety scenarios.<n> MMSafeAware includes both unsafe and over-safety subsets to assess models abilities to correctly identify unsafe content and avoid over-sensitivity that can hinder helpfulness.
  arXiv  Detail & Related papers  (2025-02-16T16:12:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.