SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation

• URL: http://arxiv.org/abs/2608.16081v1
• Date: Mon, 17 Aug 2026 04:25:58 GMT
• Title: SafeGesture: Evaluating Fine-Grained Hand Gesture Understanding in Vision-Language Models through Scenario-Conditioned Safety Interpretation
• Abstract summary: We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures.<n>Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models.<n>Results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.
• Score: 0.4779196219827507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-weight and frontier vision-language models (VLMs) perform well on general image understanding, but their ability to interpret fine-grained hand gestures in safety-critical operational contexts remains largely unexamined. We introduce SafeGesture, a benchmark that evaluates whether a model can infer scenario-appropriate safety actions from hand gestures. It pairs six HaGRID gestures with eight operational scenarios for 4,800 items and evaluates Qwen2.5-VL-7B, LLaVA-NeXT-7B, InternVL2-8B, Phi-3.5-Vision, and GPT-4o. Results reveal a perception-reasoning decoupling: GPT-4o achieves 98.4% gesture accuracy but 53.3% safety accuracy, while Qwen2.5-VL reaches 84.9% and 39.5%, yielding gaps of 45.0 and 45.4 percentage points. Four of five models rarely or never use the uncertainty label, and failure directions differ substantially across models. Accuracy also obscures label bias: a scenario-majority policy with no visual input reaches 58.3%, above every evaluated model, while only GPT-4o exceeds this prior under macro-F1. Visual input improves safety accuracy by 11.2 to 30.2 percentage points, but providing the ground-truth gesture as text improves performance by only 0.4 to 3.2 points, and no model exceeds 56.2%. These results indicate that the main bottleneck is scenario-conditioned safety reasoning rather than gesture recognition.

Related papers

• EviSafe: Evidence-Grounded Safety Evaluation for Vision-Language Models [3.659020283346239]
  We introduce EviSafe, an evidence-grounded framework for VLM safety.<n>It jointly evaluates natural user-facing behavior, explicit grounding in textual and visual evidence, and behavioral sensitivity to counterfactual changes.<n>EviSafeBench instantiates the framework as a controlled benchmark with 1,181 gold image-text scenarios.
  arXiv  Detail & Related papers  (2026-08-24T14:32:25Z)
• Benchmarking the Robustness of Autonomous Driving to Environmental Illusions: A Lane Perception Perspective [76.02772556944073]
  Environmental illusions are naturally existing yet overlooked phenomena in real-world driving environments.<n>They can disturb visual perception, leading to misinterpretation of the scene and posing serious safety risks to autonomous driving systems.<n>We introduce the first benchmark, LanEvil++, for evaluating the robustness of lane perception under environmental illusions.
  arXiv  Detail & Related papers  (2026-07-07T03:15:00Z)
• When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models [5.03712104249503]
  Safety benchmark scores provide incomplete evidence of deployment readiness.<n> aligned language models often adhere to rigid rules even when a situational update flips which action is safe.<n>We test 12 models across a safety benchmark and two commonsense controls using paired variants where the nominally safe action produces harm.
  arXiv  Detail & Related papers  (2026-05-27T02:09:30Z)
• Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries [0.0]
  Large language models are trained to refuse harmful requests, but can they accurately predict when they will refuse before responding?<n>We investigate this question through a systematic study where models first predict their refusal behavior, then respond in a fresh context.<n>Using signal detection theory (SDT), we find that all models exhibit high introspective sensitivity (d' = 2.4-3.5), but sensitivity drops substantially at safety boundaries.
  arXiv  Detail & Related papers  (2026-03-31T20:49:34Z)
• VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering [14.756677328512907]
  VisualLeakBench is an evaluation suite to audit LVLMs against OCR Injection and Contextual PII Leakage.<n>We use 1,000 synthetically generated adversarial images with 8 PII types, validated on 50 in-the-wild (IRL) real-world screenshots.<n>We release our dataset and code for reproducible robustness and safety evaluation of deployment-relevant vision-language systems.
  arXiv  Detail & Related papers  (2026-03-11T05:47:24Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5 [101.4233736714284]
  Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have driven major gains in reasoning, perception, and generation across language and vision.<n>We present an integrated safety evaluation of six frontier models--GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5--assessing each across language, vision-language, and image generation.
  arXiv  Detail & Related papers  (2026-01-15T15:52:52Z)
• DUAL-Bench: Measuring Over-Refusal and Robustness in Vision-Language Models [59.45605332033458]
  Safety mechanisms can backfire, causing over-refusal, where models decline benign requests out of excessive caution.<n>No existing benchmark has systematically addressed over-refusal in the visual modality.<n>This setting introduces unique challenges, such as dual-use cases where an instruction is harmless, but the accompanying image contains harmful content.
  arXiv  Detail & Related papers  (2025-10-12T23:21:34Z)
• SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Models [66.71948519280669]
  Multimodal Large Reasoning Models (MLRMs) demonstrate impressive crossmodal reasoning but often amplify safety risks under adversarial prompts.<n> Existing defenses mainly act at the output level and do not constrain the reasoning process, leaving models to implicit risks.<n>We propose SaFeR-VLM, which integrates four components and supports dynamic and interpretable safety decisions beyond surface-level filtering.
  arXiv  Detail & Related papers  (2025-10-08T10:39:12Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.