Towards Mechanistic Defenses Against Typographic Attacks in CLIP

• URL: http://arxiv.org/abs/2508.20570v1
• Date: Thu, 28 Aug 2025 09:08:30 GMT
• Title: Towards Mechanistic Defenses Against Typographic Attacks in CLIP
• Abstract summary: We analyze how CLIP vision encoders behave under typographic attacks.<n>We introduce a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit.<n>We release a family of dyslexic CLIP models which are significantly more robust against typographic attacks.
• Score: 23.69564867168339
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Typographic attacks exploit multi-modal systems by injecting text into images, leading to targeted misclassifications, malicious content generation and even Vision-Language Model jailbreaks. In this work, we analyze how CLIP vision encoders behave under typographic attacks, locating specialized attention heads in the latter half of the model's layers that causally extract and transmit typographic information to the cls token. Building on these insights, we introduce a method to defend CLIP models against typographic attacks by selectively ablating a typographic circuit, consisting of attention heads. Without requiring finetuning, our method improves performance by up to 19.6% on a typographic variant of ImageNet-100, while reducing standard ImageNet-100 accuracy by less than 1%. Notably, our training-free approach remains competitive with current state-of-the-art typographic defenses that rely on finetuning. To this end, we release a family of dyslexic CLIP models which are significantly more robust against typographic attacks. These models serve as suitable drop-in replacements for a broad range of safety-critical applications, where the risks of text-based manipulation outweigh the utility of text recognition.

Related papers

• Towards Robustness against Typographic Attack with Training-free Concept Localization [42.575178574443946]
  Contrastive Language-Image Pretraining models (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs)<n>CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics.<n>This issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving.<n>We propose a novel, training-free mechanistic interpretability method to achieve interpretable and effective robustness against TA.
  arXiv  Detail & Related papers  (2026-07-02T17:55:24Z)
• Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models [89.0460992131069]
  We propose AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model.<n>Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss.<n>We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively.
  arXiv  Detail & Related papers  (2026-04-13T14:54:25Z)
• Reading Between the Pixels: An Inscriptive Jailbreak Attack on Text-to-Image Models [31.243185346527255]
  Modern text-to-image (T2I) models can now render legible, paragraph-length text, enabling a fundamentally new class of misuse.<n>We identify and formalize the inscriptive jailbreak, where an adversary coerces a T2I system into generating images containing harmful textual payloads.<n>We propose Etch, a black-box attack framework that decomposes the adversarial prompt into three functional layers.
  arXiv  Detail & Related papers  (2026-04-07T13:16:07Z)
• A Generative Adversarial Approach to Adversarial Attacks Guided by Contrastive Language-Image Pre-trained Model [12.15621649989295]
  A generative adversarial attack method is proposed that uses the CLIP model to create highly effective and visually imperceptible adversarial perturbations.<n>Our approach integrates the concentrated perturbation strategy from Saliency-based Auto-Encoder with the dissimilar text embeddings similar to Generative Adversarial Multi-Object Scene Attacks (GAMA)
  arXiv  Detail & Related papers  (2025-11-03T08:02:48Z)
• Leveraging Hierarchical Image-Text Misalignment for Universal Fake Image Detection [58.927873049646024]
  We show that fake images cannot be properly aligned with corresponding captions compared to real images.<n>We propose a simple yet effective ITEM by leveraging the image-text misalignment in a joint visual-language space as discriminative clues.
  arXiv  Detail & Related papers  (2025-11-01T06:51:14Z)
• SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models [1.0260880679794957]
  Typographic attacks exploit the interplay between text and visual content in multimodal foundation models.<n>We introduce SCAM, the largest and most diverse dataset of real-world typographic attack images to date.<n>Our findings indicate that typographic attacks remain effective against state-of-the-art Large Vision-Language Models.
  arXiv  Detail & Related papers  (2025-04-07T10:01:38Z)
• Web Artifact Attacks Disrupt Vision Language Models [61.59021920232986]
  Vision-language models (VLMs) are trained on large-scale, lightly curated web datasets.<n>They learn unintended correlations between semantic concepts and unrelated visual signals.<n>Prior work has weaponized these correlations as an attack vector to manipulate model predictions.<n>We introduce "artifact-based" attacks: a novel class of manipulations that mislead models using both non-matching text and graphical elements.
  arXiv  Detail & Related papers  (2025-03-17T18:59:29Z)
• Undermining Image and Text Classification Algorithms Using Adversarial Attacks [0.0]
  Our study addresses the gap by training various machine learning models and using GANs and SMOTE to generate additional data points aimed at attacking text classification models. Our experiments reveal a significant vulnerability in classification models. Specifically, we observe a 20 % decrease in accuracy for the top-performing text classification models post-attack, along with a 30 % decrease in facial recognition accuracy.
  arXiv  Detail & Related papers  (2024-11-03T18:44:28Z)
• Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models [64.67721492968941]
  We propose a Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR) framework. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness. Our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques.
  arXiv  Detail & Related papers  (2024-10-29T07:15:09Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)
• Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks [58.10730906004818]
  Typographic attacks, adding misleading text to images, can deceive vision-language models (LVLMs)<n>Our experiments show these attacks significantly reduce classification performance by up to 60%.
  arXiv  Detail & Related papers  (2024-02-01T14:41:20Z)
• Defense-Prefix for Preventing Typographic Attacks on CLIP [14.832208701208414]
  Some adversarial attacks fool a model into false or absurd classifications. We introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words "robust" against typographic attacks. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model.
  arXiv  Detail & Related papers  (2023-04-10T11:05:20Z)
• Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks [52.26631767748843]
  We propose ROCLIP, the first effective method for robust pre-training multimodal vision-language models against targeted data poisoning and backdoor attacks. ROCLIP effectively breaks the association between poisoned image-caption pairs by considering a relatively large and varying pool of random captions. Our experiments show that ROCLIP renders state-of-the-art targeted data poisoning and backdoor attacks ineffective during pre-training CLIP models.
  arXiv  Detail & Related papers  (2023-03-13T04:49:46Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.