Towards Robustness against Typographic Attack with Training-free Concept Localization

• URL: http://arxiv.org/abs/2607.02494v1
• Date: Thu, 02 Jul 2026 17:55:24 GMT
• Title: Towards Robustness against Typographic Attack with Training-free Concept Localization
• Abstract summary: Contrastive Language-Image Pretraining models (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs)<n>CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics.<n>This issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving.<n>We propose a novel, training-free mechanistic interpretability method to achieve interpretable and effective robustness against TA.
• Score: 42.575178574443946
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Models trained via Contrastive Language-Image Pretraining (CLIP) serve as the foundational vision encoders for most modern Large Vision Language Models (LVLMs). Despite their widespread adoption, CLIP models exhibit a critical yet underexplored failure mode: irrelevant text appearing within images confounds visual representations, biasing them toward lexical meaning rather than true visual semantics. This robustness issue, commonly described as a Typographic Attack (TA), exposes a vulnerability that poses a significant risk to safety-critical applications such as autonomous driving. To achieve interpretable and effective robustness against TA, we propose a novel, training-free mechanistic interpretability method. Our method provides sampling-based interpretations of hidden state representations and quantitatively attributes semantic versus lexical focus to individual attention heads. Through probabilistic analysis and circuit mining, we isolate specific Vision Transformer (ViT) components that disproportionately encode lexical information, thereby identifying the mechanistic source of TA. We further show that simple interventions applied directly to the identified circuits, without any additional training, can substantially improve robustness against Typographic Attacks in object classification. These interventions, such as selective adjustment of attention weights, also outperform both supervised and training-free defense methods. Our experiments demonstrate that applying the proposed intervention to the vision encoders of several state-of-the-art LVLMs yields substantial gains in Visual Question Answering accuracy under Typographic Attack interference on RIO-Bench. These results confirm both the efficacy and the generalizability of our mechanistic approach. Code is released at https://github.com/Liu-524/SamplingTAR.

Related papers

• Can Vision-Language Models Reason about AI Edits in Images? [78.04125956307838]
  Vision-Language Models (VLMs) offer a promising alternative due to their strong visual understanding and reasoning capabilities.<n>We propose a GRPO-based training framework that utilizes simple accuracy and format rewards.<n>Our approach achieves competitive detection and localization performance compared to state-of-the-art image forgery detectors.
  arXiv  Detail & Related papers  (2026-07-30T16:23:40Z)
• DE-FIVE: Detecting Malicious Image Prompts via Fourier Features and Image Vector Embeddings [0.870967746482596]
  We introduce DE-FIVE, a training-free framework for detecting malicious image prompts.<n>We develop a hybrid detection strategy consisting of a black-box detector and a white-box detector.<n>Extensive experiments demonstrate that the proposed framework consistently outperforms state-of-the-art baselines against malicious image prompts.
  arXiv  Detail & Related papers  (2026-06-22T02:42:38Z)
• Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models [17.259725776748482]
  Existing adversarial training methods for robust fine-tuning largely overlook the role of language in enhancing visual robustness.<n>We propose Quality Text-guided Adversarial Fine-Tuning (QT-AFT), which leverages high-quality captions during training to guide adversarial examples away from diverse semantics present in images.<n> QT-AFT achieves state-of-the-art zero-shot adversarial robustness and clean accuracy, evaluated across 16 zero-shot datasets.
  arXiv  Detail & Related papers  (2025-07-22T06:13:30Z)
• Prompt-driven Transferable Adversarial Attack on Person Re-Identification with Attribute-aware Textual Inversion [17.18411620606476]
  We introduce the Attribute-aware Prompt Attack (AP-Attack) to disrupt fine-grained semantic features of pedestrian images.<n>AP-Attack achieves state-of-the-art transferability, significantly outperforming previous methods by 22.9% on mean Drop Rate.
  arXiv  Detail & Related papers  (2025-02-27T02:32:58Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)
• SA-Attack: Improving Adversarial Transferability of Vision-Language Pre-training Models via Self-Augmentation [56.622250514119294]
  In contrast to white-box adversarial attacks, transfer attacks are more reflective of real-world scenarios. We propose a self-augment-based transfer attack method, termed SA-Attack.
  arXiv  Detail & Related papers  (2023-12-08T09:08:50Z)
• Counterfactual Image Generation for adversarially robust and interpretable Classifiers [1.3859669037499769]
  We propose a unified framework leveraging image-to-image translation Generative Adrial Networks (GANs) to produce counterfactual samples. This is achieved by combining the classifier and discriminator into a single model that attributes real images to their respective classes and flags generated images as "fake" We show how the model exhibits improved robustness to adversarial attacks, and we show how the discriminator's "fakeness" value serves as an uncertainty measure of the predictions.
  arXiv  Detail & Related papers  (2023-10-01T18:50:29Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.