SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues

• URL: http://arxiv.org/abs/2603.19092v1
• Date: Thu, 19 Mar 2026 16:18:00 GMT
• Title: SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cues
• Abstract summary: We study whether multimodal safety behavior in vision-language models can be steered by simple semantic cues.<n>We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content.
• Score: 50.928203120795786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) are increasingly deployed in real-world and embodied settings where safety decisions depend on visual context. However, it remains unclear which visual evidence drives these judgments. We study whether multimodal safety behavior in VLMs can be steered by simple semantic cues. We introduce a semantic steering framework that applies controlled textual, visual, and cognitive interventions without changing the underlying scene content. To evaluate these effects, we propose SAVeS, a benchmark for situational safety under semantic cues, together with an evaluation protocol that separates behavioral refusal, grounded safety reasoning, and false refusals. Experiments across multiple VLMs and an additional state-of-the-art benchmark show that safety decisions are highly sensitive to semantic cues, indicating reliance on learned visual-linguistic associations rather than grounded visual understanding. We further demonstrate that automated steering pipelines can exploit these mechanisms, highlighting a potential vulnerability in multimodal safety systems.

Related papers

• HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models [87.35765363039638]
  We introduce HazardArena, a benchmark designed to evaluate semantic safety in Vision-Language-Action models.<n>We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts.<n>We propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge.
  arXiv  Detail & Related papers  (2026-04-14T08:32:02Z)
• Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory [10.434155461003387]
  We present MM-SafetyBench++, a benchmark designed for contextual safety evaluation.<n>For each unsafe image-text pair, we construct a corresponding safe counterpart through minimal modifications.<n>We also introduce EchoSafe, a training-free framework that maintains a self-reflective memory bank to accumulate and retrieve safety insights.
  arXiv  Detail & Related papers  (2026-03-16T18:32:26Z)
• Vision and language: Novel Representations and Artificial intelligence for Driving Scene Safety Assessment and Autonomous Vehicle Planning [2.1379801460200416]
  Vision-language models (VLMs) have emerged as powerful representation learning systems that align visual observations with natural language concepts.<n>This paper investigates how vision-language representations support driving scene safety assessment and decision-making when integrated into perception, prediction, and planning pipelines.
  arXiv  Detail & Related papers  (2026-02-07T20:04:21Z)
• Semantic Misalignment in Vision-Language Models under Perceptual Degradation [2.9140696506330723]
  We study semantic misalignment in Vision-Language Models (VLMs) under controlled degradation of visual perception.<n>We observe severe failures in downstream VLM behavior, including hallucinated object mentions, omission of safety-critical entities, and inconsistent safety judgments.<n>Our results reveal a clear disconnect between pixel-level robustness and multimodal semantic reliability, highlighting a critical limitation of current VLM-based systems.
  arXiv  Detail & Related papers  (2026-01-13T09:13:05Z)
• Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations [94.62792643569567]
  This work systematically investigates the role of speaker emotion.<n>We construct a dataset of malicious speech instructions expressed across multiple emotions and intensities, and evaluate several state-of-the-art LALMs.<n>Our results reveal substantial safety inconsistencies: different emotions elicit varying levels of unsafe responses, and the effect of intensity is non-monotonic, with medium expressions often posing the greatest risk.
  arXiv  Detail & Related papers  (2025-10-19T15:41:25Z)
• Self-Aware Safety Augmentation: Leveraging Internal Semantic Understanding to Enhance Safety in Vision-Language Models [21.961325147038867]
  Large vision-language models (LVLMs) are vulnerable to harmful input compared to their language-only backbones.<n>We define these capabilities as safety perception, semantic understanding, and alignment for linguistic expression.<n>Motivated by these findings, we propose textbfSelf-Aware Safety Augmentation (SASA), a technique that projects informative semantic representations onto earlier safety-oriented layers.
  arXiv  Detail & Related papers  (2025-07-29T09:48:57Z)
• HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model [58.12612140992874]
  We introduce a holistic safety dataset and benchmark, textbfHoliSafe, that spans all five safe/unsafe image-text combinations.<n>We also propose a novel modular framework for enhancing VLM safety with a visual guard module (VGM) designed to assess the harmfulness of input images.<n> Experiments show that Safe-VLM with VGM, trained on our HoliSafe, achieves state-of-the-art safety performance across multiple VLM benchmarks.
  arXiv  Detail & Related papers  (2025-06-05T07:26:34Z)
• Multimodal Situational Safety [73.63981779844916]
  We present the first evaluation and analysis of a novel safety challenge termed Multimodal Situational Safety.<n>For an MLLM to respond safely, whether through language or action, it often needs to assess the safety implications of a language query within its corresponding visual context.<n>We develop the Multimodal Situational Safety benchmark (MSSBench) to assess the situational safety performance of current MLLMs.
  arXiv  Detail & Related papers  (2024-10-08T16:16:07Z)
• SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering [56.92068213969036]
  Safety alignment is indispensable for Large Language Models (LLMs) to defend threats from malicious instructions.<n>Recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue.<n>We propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns.
  arXiv  Detail & Related papers  (2024-08-21T10:01:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.