Localization then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack

• URL: http://arxiv.org/abs/2605.25194v1
• Date: Sun, 24 May 2026 17:51:34 GMT
• Title: Localization then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack
• Abstract summary: Adversarial images pose a severe security threat to multimodal large language models through prompt injection.<n>We show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens.<n>We propose Gradient Token Masking (GTM), which localizes these tokens via gradient analysis and neutralizes them through masking.
• Score: 78.02110947708535
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Adversarial images pose a severe security threat to multimodal large language models through prompt injection. Existing defenses largely lack a principled understanding of the underlying mechanisms and struggle to balance efficiency and defense utility. In this work, we show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens. Based on this insight, we propose Gradient Token Masking (GTM), which localizes these tokens via gradient analysis and neutralizes them through masking. We find that attribution based on the first generated token's output probability fails when attacks preserve the predicted token. To overcome this, GTM utilizes the Hidden-State Gradient Norm score for generation-influence attribution under adversarial inputs. We prove that its ranking is consistent with that of the full adversarial loss gradient, providing a theoretical guarantee for accurate localization. Our method requires only a single forward-backward pass to identify and zero out a small number of high-scoring tokens, effectively disrupting the adversarial attack path. Extensive experiments on prompt injection and multimodal jailbreak attacks demonstrate that our approach reduces attack success rates (ASR) to near zero while preserving model utility with negligible computational overhead.

Related papers

• Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring [13.497048408038935]
  Large Vision-Language Models (LVLMs) are vulnerable to a growing array of multimodal jailbreak attacks.<n>Current anomaly-detection methods tend to confuse novel benign inputs with malicious ones, leading to unreliable over-rejection.<n>We propose Representational Contrastive Scoring (RCS), a framework built on a key insight: the most potent safety signals reside within the LVLM's own internal representations.
  arXiv  Detail & Related papers  (2025-12-12T22:31:38Z)
• Matching Ranks Over Probability Yields Truly Deep Safety Alignment [8.692532301315135]
  Recent work proposed a supervised fine-tuning (SFT) defense using data augmentation to achieve a enquotedeep safety alignment.<n>We show in this work that the "deep" safety alignment produced by such an approach is in fact not very deep.<n>We propose a new perspective on achieving deep safety alignment by matching the token ranks of the target distribution, rather than their probabilities.
  arXiv  Detail & Related papers  (2025-12-05T08:22:27Z)
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model [91.48868589442837]
  We present the Deadlock Attack, a resource exhaustion method that hijacks an LRM's generative control flow.<n>Our method achieves a 100% attack success rate across four advanced LRMs.
  arXiv  Detail & Related papers  (2025-10-12T07:42:57Z)
• Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations [10.746349111023964]
  We introduce a novel approach that injects the IH signal into the intermediate token representations within the network.<n>Our method augments these representations with layer-specific trainable embeddings that encode the privilege information.<n>Our evaluations across multiple models and training methods reveal that our proposal yields between $1.6times$ and $9.2times$ reduction in attack success rate.
  arXiv  Detail & Related papers  (2025-05-25T00:01:39Z)
• Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries [22.24239212756129]
  We find that simply appending multiple end of sequence (eos) tokens can cause a phenomenon we call context segmentation.<n>We propose a straightforward method to BOOST jailbreak attacks by appending eos tokens.
  arXiv  Detail & Related papers  (2024-05-31T07:41:03Z)
• Boosting Jailbreak Attack with Momentum [5.047814998088682]
  Large Language Models (LLMs) have achieved remarkable success across diverse tasks, yet they remain vulnerable to adversarial attacks.<n>textbfAccelerated GbfCG (textbfMAC) attack integrates a momentum term into the gradient to boost and stabilize the random search for tokens in adversarial prompts.
  arXiv  Detail & Related papers  (2024-05-02T12:18:14Z)
• Patch-wise++ Perturbation for Adversarial Targeted Attacks [132.58673733817838]
  We propose a patch-wise iterative method (PIM) aimed at crafting adversarial examples with high transferability. Specifically, we introduce an amplification factor to the step size in each iteration, and one pixel's overall gradient overflowing the $epsilon$-constraint is properly assigned to its surrounding regions. Compared with the current state-of-the-art attack methods, we significantly improve the success rate by 35.9% for defense models and 32.7% for normally trained models.
  arXiv  Detail & Related papers  (2020-12-31T08:40:42Z)
• GreedyFool: Distortion-Aware Sparse Adversarial Attack [138.55076781355206]
  Modern deep neural networks (DNNs) are vulnerable to adversarial samples. Sparse adversarial samples can fool the target model by only perturbing a few pixels. We propose a novel two-stage distortion-aware greedy-based method dubbed as "GreedyFool"
  arXiv  Detail & Related papers  (2020-10-26T17:59:07Z)
• Patch-wise Attack for Fooling Deep Neural Network [153.59832333877543]
  We propose a patch-wise iterative algorithm -- a black-box attack towards mainstream normally trained and defense models. We significantly improve the success rate by 9.2% for defense models and 3.7% for normally trained models on average.
  arXiv  Detail & Related papers  (2020-07-14T01:50:22Z)
• A Self-supervised Approach for Adversarial Robustness [105.88250594033053]
  Adversarial examples can cause catastrophic mistakes in Deep Neural Network (DNNs) based vision systems. This paper proposes a self-supervised adversarial training mechanism in the input space. It provides significant robustness against the textbfunseen adversarial attacks.
  arXiv  Detail & Related papers  (2020-06-08T20:42:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.