Adversarial attacks against Modern Vision-Language Models

• URL: http://arxiv.org/abs/2603.16960v2
• Date: Sun, 22 Mar 2026 18:55:10 GMT
• Title: Adversarial attacks against Modern Vision-Language Models
• Abstract summary: We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions.<n>We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks.
• Score: 51.56484100374058
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.

Related papers

• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• Security Assessment and Mitigation Strategies for Large Language Models: A Comprehensive Defensive Framework [0.6015898117103067]
  Large Language Models increasingly power critical infrastructure from healthcare to finance, yet their vulnerability to adversarial manipulation threatens system integrity and user safety.<n>This research addresses this gap by establishing a standardized vulnerability assessment framework and developing a multi-layered defensive system to protect against identified threats.
  arXiv  Detail & Related papers  (2026-03-17T20:32:06Z)
• OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequences [64.01706941950489]
  Current safety paradigms primarily target malicious intent or situational violations.<n>We propose shifting the safety frontier toward consequence-driven safety, a paradigm essential for the robust deployment of autonomous and embodied agents.<n>We develop the Consequence-Aware Safety Policy Optimization (CASPO) framework, which integrates the model's intrinsic reasoning as a dynamic reference for token-level self-distillation rewards.
  arXiv  Detail & Related papers  (2026-03-10T14:16:43Z)
• Multi-Faceted Attack: Exposing Cross-Model Vulnerabilities in Defense-Equipped Vision-Language Models [54.61181161508336]
  We introduce Multi-Faceted Attack (MFA), a framework that exposes general safety vulnerabilities in leading defense-equipped Vision-Language Models (VLMs)<n>The core component of MFA is the Attention-Transfer Attack (ATA), which hides harmful instructions inside a meta task with competing objectives.<n>MFA achieves a 58.5% success rate and consistently outperforms existing methods.
  arXiv  Detail & Related papers  (2025-11-20T07:12:54Z)
• AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models [60.39655329875822]
  Vision-Language-Action (VLA) models enable robots to interpret natural-language instructions and perform diverse tasks.<n>Despite growing interest in attacking such models, the effectiveness of existing techniques remains unclear.<n>We propose AttackVLA, a unified framework that aligns with the VLA development lifecycle.
  arXiv  Detail & Related papers  (2025-11-15T10:30:46Z)
• Agentic UAVs: LLM-Driven Autonomy with Integrated Tool-Calling and Cognitive Reasoning [3.4643961367503575]
  Existing UAV frameworks lack context-aware reasoning, autonomous decision-making, and ecosystem-level integration.<n>This paper introduces the Agentic UAVs framework, a five-layer architecture (Perception, Reasoning, Action, Integration, Learning)<n>A ROS2 and Gazebo-based prototype integrates YOLOv11 object detection with GPT-4 reasoning and local Gemma-3 deployment.
  arXiv  Detail & Related papers  (2025-09-14T08:46:40Z)
• Differential Robustness in Transformer Language Models: Empirical Evaluation Under Adversarial Text Attacks [3.3772986620114387]
  RoBERTa-Base and FlanT5 demonstrated remarkable resilience, maintaining accuracy even when subjected to sophisticated attacks.<n>BERT-Base showed considerable vulnerability, with TextFooler achieving a 93.75% success rate in reducing model accuracy from 48% to just 3%.
  arXiv  Detail & Related papers  (2025-09-05T21:43:06Z)
• An Automated Attack Investigation Approach Leveraging Threat-Knowledge-Augmented Large Language Models [17.220143037047627]
  Advanced Persistent Threats (APTs) compromise high-value systems to steal data or disrupt operations.<n>Existing methods suffer from poor platform generality, limited generalization to evolving tactics, and an inability to produce analyst-ready reports.<n>We present an LLM-empowered attack investigation framework augmented with a dynamically adaptable Kill-Chain-aligned threat knowledge base.
  arXiv  Detail & Related papers  (2025-09-01T08:57:01Z)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models [92.79804303337522]
  Vision-Language Models (VLMs) may still be vulnerable to safety alignment issues.<n>We introduce MLAI, a novel jailbreak framework that leverages scenario-aware image generation for semantic alignment.<n>Extensive experiments demonstrate MLAI's significant impact, achieving attack success rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2.
  arXiv  Detail & Related papers  (2024-11-27T02:40:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.