GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models

• URL: http://arxiv.org/abs/2607.19683v1
• Date: Wed, 22 Jul 2026 02:33:56 GMT
• Title: GhostPrompt: Cross-Image Adversarial Prompt for Vision-Language Models
• Abstract summary: GhostPrompt is an adversarial prompt that is optimized once and reused to steer VLM outputs toward attacker-specified responses across diverse images.<n>Our method achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by 70%.
• Score: 19.133064005999838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) are known to be vulnerable to adversarial attacks, where subtle perturbations to images or texts induce erroneous outputs. However, most text-based attacks are adapted from language-model-centric methods, in which the visual input is fixed during optimization, resulting in adversarial prompts that are tied to specific images and thus limiting their attack effectiveness. To this end, we first introduce a new research perspective: cross-image transferability for adversarial prompts. We then propose GhostPrompt, an adversarial prompt that is optimized once and reused to steer VLM outputs toward attacker-specified responses across diverse images. GhostPrompt employs a joint optimization that distills image-invariant adversarial features into the prompt by "worst-case" generation. Specifically, it alternates between constructing hard visual conditions for the current prompt and updating the prompt to remain effective under these conditions. Extensive experiments on prevalent VLMs verify that \ourmethod achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by ~70%. Our code is avalable at https://github.com/Ye-ze-yu/GhostPrompt.

Related papers

• Adversarial Prompt Injection Attack on Multimodal Large Language Models [21.995939723909796]
  multimodal large language models (MLLMs) are increasingly deployed in real-world applications.<n>Existing prompt injection methods rely on textual prompts or perceptible visual prompts that are observable by human users.<n>In this work, we study imperceptible visual prompt injection against powerful closed-source MLLMs.
  arXiv  Detail & Related papers  (2026-03-31T08:22:07Z)
• An Image Is Worth Ten Thousand Words: Verbose-Text Induction Attacks on VLMs [48.05423013052023]
  This paper proposes a novel verbose-text induction attack (VTIA) to inject imperceptible adversarial perturbations into benign images.<n>We first perform adversarial prompt search, employing reinforcement learning strategies to automatically identify adversarial prompts.<n>We then conduct vision-aligned perturbation optimization to craft adversarial examples on input images, maximizing the similarity between the perturbed image's visual embeddings and those of the adversarial prompt.
  arXiv  Detail & Related papers  (2025-11-20T09:03:43Z)
• PromptFlare: Prompt-Generalized Defense via Cross-Attention Decoy in Diffusion-Based Inpainting [25.24109316946351]
  We propose PromptFlare, a novel adversarial protection method designed to protect images from malicious modifications facilitated by diffusion-based inpainting models.<n>Our approach exploits the intrinsic properties of prompt embeddings and injects adversarial noise to suppress the sampling process.<n>Experiments on the EditBench dataset demonstrate that our method achieves state-of-the-art performance across various metrics.
  arXiv  Detail & Related papers  (2025-08-22T08:42:46Z)
• ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models [67.75439511654078]
  Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses.<n>They face the persistent challenge of hallucination, which introduces practical weaknesses and raises concerns about their reliable deployment in real-world applications.<n>We propose ONLY, a training-free decoding approach that requires only a single query and a one-layer intervention during decoding, enabling efficient real-time deployment.
  arXiv  Detail & Related papers  (2025-07-01T16:01:08Z)
• Enhancing Adversarial Robustness of Vision Language Models via Adversarial Mixture Prompt Tuning [14.714776642137247]
  Adversarial Mixture Prompt Tuning (AMPT) aims to learn mixture text prompts to obtain more robust text features.<n>A series of experiments show that our method can achieve better adversarial robustness than state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-23T06:04:15Z)
• FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weighting [18.708185548091716]
  FRAP is a simple, yet effective approach based on adaptively adjusting the per-token prompt weights.<n>We show FRAP generates images with significantly higher prompt-image alignment to prompts from complex datasets.<n>We also explore combining FRAP with prompt rewriting LLM to recover their degraded prompt-image alignment.
  arXiv  Detail & Related papers  (2024-08-21T15:30:35Z)
• Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt [60.54666043358946]
  This paper introduces the Bi-Modal Adversarial Prompt Attack (BAP), which executes jailbreaks by optimizing textual and visual prompts cohesively. In particular, we utilize a large language model to analyze jailbreak failures and employ chain-of-thought reasoning to refine textual prompts.
  arXiv  Detail & Related papers  (2024-06-06T13:00:42Z)
• Dynamic Prompt Optimizing for Text-to-Image Generation [63.775458908172176]
  We introduce the textbfPrompt textbfAuto-textbfEditing (PAE) method to improve text-to-image generative models. We employ an online reinforcement learning strategy to explore the weights and injection time steps of each word, leading to the dynamic fine-control prompts.
  arXiv  Detail & Related papers  (2024-04-05T13:44:39Z)
• Adversarial Prompt Tuning for Vision-Language Models [86.5543597406173]
  Adversarial Prompt Tuning (AdvPT) is a technique to enhance the adversarial robustness of image encoders in Vision-Language Models (VLMs) We demonstrate that AdvPT improves resistance against white-box and black-box adversarial attacks and exhibits a synergistic effect when combined with existing image-processing-based defense techniques.
  arXiv  Detail & Related papers  (2023-11-19T07:47:43Z)
• Progressive Visual Prompt Learning with Contrastive Feature Re-formation [15.385630262368661]
  We propose a new Progressive Visual Prompt (ProVP) structure to strengthen the interactions among prompts of different layers. Our ProVP could effectively propagate the image embeddings to deep layers and behave partially similar to an instance adaptive prompt method. To the best of our knowledge, we are the first to demonstrate the superior performance of visual prompts in V-L models to previous prompt-based methods in downstream tasks.
  arXiv  Detail & Related papers  (2023-04-17T15:54:10Z)
• Bayesian Prompt Learning for Image-Language Model Generalization [64.50204877434878]
  We use the regularization ability of Bayesian methods to frame prompt learning as a variational inference problem. Our approach regularizes the prompt space, reduces overfitting to the seen prompts and improves the prompt generalization on unseen prompts. We demonstrate empirically on 15 benchmarks that Bayesian prompt learning provides an appropriate coverage of the prompt space.
  arXiv  Detail & Related papers  (2022-10-05T17:05:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.