Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks

• URL: http://arxiv.org/abs/2608.01043v1
• Date: Sun, 02 Aug 2026 07:00:49 GMT
• Title: Decoy Images Amplify Caption-Mediated Defenses Against Encoded Jailbreaks
• Abstract summary: We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models.<n>A caption-mediated defense (ECSO) that leaves ASR essentially unchanged on text-only encoded input drops it by up to $73$pp once a content-free decoy is attached.<n>We frame this as an observation about pipeline interaction, not as a robust defense.
• Score: 7.766839238117721
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR). The operative change is in the defense pipeline, not in the image. Across five frontier VLMs, two encoded-attack families, and three black-box defenses, a caption-mediated defense (ECSO) that leaves ASR essentially unchanged on text-only encoded input drops it by up to $73$pp once a content-free decoy is attached; every non-saturated contrast is significant under exact McNemar tests. We advance two hypotheses for this pattern, supported by indirect evidence rather than pipeline introspection, since a black-box threat model precludes inspecting vendor internals: caption-mediated defenses branch on image presence, and intrinsic image-side safety engages on image-resident content. Three controls constrain the explanation. Blank-canvas and natural-photograph decoys reproduce the effect on every model, implicating image presence rather than content; the effect replicates on three open-weight VLMs served with no moderation layer, so it is not a vendor-filtering artifact; and a non-symbolic, meaning-based encoder reproduces it, so it is not specific to symbolic obfuscation. Attaching a decoy unconditionally is not deployable --- it raises benign refusal to $20$--$79\%$, an inflation of $+10$ to $+67$pp --- but gating attachment on a lightweight encoded-input detector returns benign refusal to the text baseline while preserving the safety gain wherever the detector fires, making detector recall the binding constraint. Under adaptive attacks that target the caption-mediated re-check, the effect degrades but holds. We frame this as an observation about pipeline interaction, not as a robust defense.

Related papers

• DiSCO: Defending text-to-image generation through distribution-guided contrastive prompt optimization [65.89030524420933]
  DiSCO is a zero-shot, strictly black-box defense that operates entirely at the prompt level as a plug-and-play module.<n>We demonstrate that DiSCO consistently enhances the safety of both undefended and defended models.<n>As a black-box, architecture-agnostic module, DiSCO can be readily applied to any text-to-image system.
  arXiv  Detail & Related papers  (2026-08-17T19:13:55Z)
• Recover, Decode, Reguard: Guard-Agnostic Defense Amplification againstEncoded VLM Jailbreaks [7.287314790466206]
  We build a guard-agnostic recover-and-decode amplifier for vision-language models.<n>We evaluate it against an ensemble of eleven attacks, scoring a behavior as broken if any succeeds.<n>This exposes our central finding: an empirical safety-utility ceiling for the non-iterative recovery defenses we evaluate.
  arXiv  Detail & Related papers  (2026-07-29T07:53:35Z)
• Localization then Neutralization: Gradient-guided Token Suppression against Visual Prompt Injection Attack [78.02110947708535]
  Adversarial images pose a severe security threat to multimodal large language models through prompt injection.<n>We show that successful adversarial attacks do not rely on the entire image uniformly but instead depend on a small subset of critical image tokens.<n>We propose Gradient Token Masking (GTM), which localizes these tokens via gradient analysis and neutralizes them through masking.
  arXiv  Detail & Related papers  (2026-05-24T17:51:34Z)
• Disciplined Diffusion: Text-to-Image Diffusion Model against NSFW Generation [9.862349752373577]
  Text-to-image (T2I) diffusion models have the ability to build high-quality pictures from text prompts.<n>They pose safety concerns because they can generate offensive or disturbing imagery when provided with harmful inputs.<n>We propose Disciplined Diffusion (DDiffusion), a novel robust text-to-image diffusion that counters Not Safe For Work (NSFW) generation.
  arXiv  Detail & Related papers  (2026-05-01T21:33:06Z)
• Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs [19.511422290404138]
  multimodal jailbreak strategies fail to engage with the complex semantic structures intrinsic to visual data.<n>We introduce textbfMemJack, a framework that explicitly leverages visual semantics to orchestrate automated jailbreak attacks.<n>MemJack achieves a 71.48% ASR against Qwen3-VL-Plus, scaling to 90% under extended budgets.
  arXiv  Detail & Related papers  (2026-04-14T11:44:59Z)
• UniMark: Unified Adaptive Multi-bit Watermarking for Autoregressive Image Generators [1.7507301683087861]
  We propose a training-free, unified watermarking framework for autoregressive image generators.<n>method introduces three core components: textbf Semantic Grouping (ASG), textbfBlockwise Multi-bit.<n>Experiments on three AR models demonstrate that method achieves state-of-the-art performance in image quality (FID), watermark detection accuracy, and multi-bit message extraction.
  arXiv  Detail & Related papers  (2026-04-12T16:22:24Z)
• SemBind: Binding Diffusion Watermarks to Semantics Against Black-Box Forgery Attacks [74.76909939060833]
  Black-box forgery attacks pose outsized risk to provenance and trust.<n>We propose SemBind, a framework for latent-based watermarks that resists black-box forgery.<n>We show that SemBind-enabled anti-forgery variants markedly reduce false acceptance under black-box forgery.
  arXiv  Detail & Related papers  (2026-01-28T07:02:40Z)
• Multimodal Prompt Decoupling Attack on the Safety Filters in Text-to-Image Models [73.43013217318965]
  Multimodal Prompt Decoupling Attack (MPDA)<n>MPDA uses image modality to separate the harmful semantic components of the original unsafe prompt.<n>Visual language model generates image captions to ensure semantic consistency between the generated NSFW images and the original unsafe prompts.
  arXiv  Detail & Related papers  (2025-09-21T11:22:32Z)
• SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs [57.880467106470775]
  Attackers can inject imperceptible perturbations into the training data, causing the model to generate malicious, attacker-controlled captions.<n>We propose Semantic Reward Defense (SRD), a reinforcement learning framework that mitigates backdoor behavior without prior knowledge of triggers.<n>SRD uses a Deep Q-Network to learn policies for applying discrete perturbations to sensitive image regions, aiming to disrupt the activation of malicious pathways.
  arXiv  Detail & Related papers  (2025-06-05T08:22:24Z)
• Divide-and-Conquer Attack: Harnessing the Power of LLM to Bypass Safety Filters of Text-to-Image Models [1.5408065585641535]
  We introduce the Divide-and-Conquer Attack to circumvent the safety filters of state-of-the-art TTI models. We design attack helper prompts that effectively guide LLMs to break down an unethical drawing intent into benign descriptions. Our findings have more severe security implications than methods of manual crafting or iterative TTI model querying.
  arXiv  Detail & Related papers  (2023-12-12T10:04:43Z)
• Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots [65.302728042116]
  We investigate the emergence of objects in visual perception in the absence of any semantic annotation.<n>The resulting model has received no supervision, does not use any pre-trained features, and yet it can segment the domain of an image into multiple moving regions.<n>The resulting motion segmentation method can handle an unknown and varying number of objects in real-time.
  arXiv  Detail & Related papers  (2023-04-04T00:26:13Z)
• Mask and Restore: Blind Backdoor Defense at Test Time with Masked Autoencoder [50.1394620328318]
  Existing backdoor defense methods often require accessing a few validation data and model parameters.<n>We propose blind backdoor defense with Masked AutoEncoder (BDMAE)<n>BDMAE detects possible local triggers using image structural similarity and label consistency between the test image and MAE restorations.
  arXiv  Detail & Related papers  (2023-03-27T19:23:33Z)
• Breaking certified defenses: Semantic adversarial examples with spoofed robustness certificates [57.52763961195292]
  We present a new attack that exploits not only the labelling function of a classifier, but also the certificate generator. The proposed method applies large perturbations that place images far from a class boundary while maintaining the imperceptibility property of adversarial examples.
  arXiv  Detail & Related papers  (2020-03-19T17:59:44Z)
• Double Backpropagation for Training Autoencoders against Adversarial Attack [15.264115499966413]
  This paper focuses on the adversarial attack on autoencoders. We propose to adopt double backpropagation (DBP) to secure autoencoder such as VAE and DRAW.
  arXiv  Detail & Related papers  (2020-03-04T05:12:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.