Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models

• URL: http://arxiv.org/abs/2604.12371v2
• Date: Wed, 15 Apr 2026 02:42:08 GMT
• Title: Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models
• Abstract summary: We study typographic prompt injection attacks on vision-language models (VLMs), where adversarial text is rendered as images to bypass safety mechanisms.<n>In practice, the attack surface is heterogeneous: adversarial text appears at varying font sizes and under diverse visual conditions.
• Score: 4.577407934990345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study typographic prompt injection attacks on vision-language models (VLMs), where adversarial text is rendered as images to bypass safety mechanisms, posing a growing threat as VLMs serve as the perceptual backbone of autonomous agents, from browser automation and computer-use systems to camera-equipped embodied agents. In practice, the attack surface is heterogeneous: adversarial text appears at varying font sizes and under diverse visual conditions, while the growing ecosystem of VLMs exhibits substantial variation in vulnerability, complicating defensive approaches. Evaluating 1,000 prompts from SALAD-Bench across four VLMs, namely, GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL-4B-Instruct under varying font sizes (6--28px) and visual transformations (rotation, blur, noise, contrast changes), we find: (1) font size significantly affects attack success rate (ASR), with very small fonts (6px) yielding near-zero ASR while mid-range fonts achieve peak effectiveness; (2) text attacks are more effective than image attacks for GPT-4o (36% vs 8%) and Claude (47% vs 22%), while Qwen3-VL and Mistral show comparable ASR across modalities; (3) text-image embedding distance from two multimodal embedding models (JinaCLIP and Qwen3-VL-Embedding) shows strong negative correlation with ASR across all four models (r = -0.71 to -0.93, p < 0.01); (4) heavy degradations increase embedding distance by 10--12% and reduce ASR by 34--96%, while rotation asymmetrically affects models (Mistral drops 50%, GPT-4o unchanged). These findings highlight that model-specific robustness patterns preclude one-size-fits-all defenses and offer empirical guidance for practitioners selecting VLM backbones for agentic systems operating in adversarial environments.

Related papers

• One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbations [0.8116687935773981]
  Typographic prompt injection exploits vision language models' (VLMs) ability to read text rendered in images.<n>We show how to maximize image text embedding similarity under bounded $ell_infty$ perturbations.<n>Experiments on GPT-4o, Claude Sonnet 4.5, Mistral-Large-3, and Qwen3-VL confirm that optimization recovers readability and reduces safety aligned refusals.
  arXiv  Detail & Related papers  (2026-04-28T01:21:47Z)
• Adversarial attacks against Modern Vision-Language Models [51.56484100374058]
  We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions.<n>We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks.
  arXiv  Detail & Related papers  (2026-03-17T04:55:10Z)
• VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering [14.756677328512907]
  VisualLeakBench is an evaluation suite to audit LVLMs against OCR Injection and Contextual PII Leakage.<n>We use 1,000 synthetically generated adversarial images with 8 PII types, validated on 50 in-the-wild (IRL) real-world screenshots.<n>We release our dataset and code for reproducible robustness and safety evaluation of deployment-relevant vision-language systems.
  arXiv  Detail & Related papers  (2026-03-11T05:47:24Z)
• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models [64.56065206447788]
  Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets.<n>We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometrics.<n>We show that low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions.
  arXiv  Detail & Related papers  (2026-03-06T10:58:02Z)
• VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models [19.867040067010674]
  We introduce VERA-V, a variational inference framework that recasts multimodal jailbreak discovery as learning a joint posterior distribution over paired text-image prompts.<n>We train a lightweight attacker to approximate the posterior, allowing efficient sampling of diverse jailbreaks.<n>Experiments on HarmBench and HADES benchmarks show that VERA-V consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-20T17:12:10Z)
• On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations [52.1029745126386]
  In vision-language-action (VLA) models, robustness to real-world perturbations is critical for deployment.<n>We propose RobustVLA against perturbations in VLA inputs and outputs.<n> Experiments on LIBERO demonstrate our RobustVLA delivers absolute gains over baselines of 12.6% on the pi0 backbone and 10.4% on the OpenVLA backbone.
  arXiv  Detail & Related papers  (2025-09-26T14:42:23Z)
• Invisible Injections: Exploiting Vision-Language Models Through Steganographic Prompt Embedding [0.0]
  Vision-language models (VLMs) have revolutionized multimodal AI applications but introduce novel security vulnerabilities that remain largely unexplored.<n>We present the first comprehensive study of steganographic prompt injection attacks againstVLMs.<n>Our approach demonstrates that current VLM architectures can inadvertently extract and execute hidden prompts during normal image processing.
  arXiv  Detail & Related papers  (2025-07-30T00:34:20Z)
• REVEAL: Multi-turn Evaluation of Image-Input Harms for Vision LLM [0.098314893665023]
  We introduce the REVEAL Framework, a scalable and automated pipeline for evaluating image-input harms in Vision Large Language Models (VLLMs)<n>We extensively evaluated five state-of-the-art VLLMs, GPT-4o, Llama-3.2, Qwen2-VL, Phi3.5V, and Pixtral, across three important harm categories: sexual harm, violence, and misinformation.<n>GPT-4o demonstrated the most balanced performance as measured by our Safety-Usability Index (SUI) followed closely by Pixtral.
  arXiv  Detail & Related papers  (2025-05-07T10:09:55Z)
• VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models [84.84277196012907]
  We propose VLsI: Verbalized Layers-to-Interactions, a new VLM family in 2B and 7B model sizes.<n>We validate VLsI across ten challenging vision-language benchmarks, achieving notable performance gains (11.0% for 2B and 17.4% for 7B) over GPT-4V.
  arXiv  Detail & Related papers  (2024-12-02T18:58:25Z)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models [92.79804303337522]
  Vision-Language Models (VLMs) may still be vulnerable to safety alignment issues.<n>We introduce MLAI, a novel jailbreak framework that leverages scenario-aware image generation for semantic alignment.<n>Extensive experiments demonstrate MLAI's significant impact, achieving attack success rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2.
  arXiv  Detail & Related papers  (2024-11-27T02:40:29Z)
• Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors [31.383591942592467]
  Vision-language models (VLMs) offer innovative ways to combine visual and textual data for enhanced understanding and interaction. Patch-based adversarial attack is considered the most realistic threat model in physical vision applications. We introduce SmoothVLM, a defense mechanism rooted in smoothing techniques, to protectVLMs from the threat of patched visual prompt injectors.
  arXiv  Detail & Related papers  (2024-05-17T04:19:19Z)
• SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation [56.622250514119294]
  In contrast to white-box adversarial attacks, transfer attacks are more reflective of real-world scenarios. We propose a self-augment-based transfer attack method, termed SA-Attack.
  arXiv  Detail & Related papers  (2023-12-08T09:08:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.