QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing

• URL: http://arxiv.org/abs/2608.13119v1
• Date: Thu, 13 Aug 2026 11:47:50 GMT
• Title: QuISE: Defense against Typographic Attacks on VLMs via Query-Irrelevant Semantic Editing
• Abstract summary: Typographic attacks pose a critical threat to vision-language models.<n>QuISE is a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing.
• Score: 4.1436924146144944
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Typographic attacks pose a critical threat to vision-language models (VLMs) by injecting misleading text into images and causing models to rely on adversarial textual cues rather than visual evidence. Existing defenses often require model-specific modifications, additional training, or access to internal model components, limiting their applicability to modern closed-source VLMs. In this paper, we propose QuISE, a model-agnostic, training-free black-box defense based on query-irrelevant semantic editing. QuISE first identifies text regions likely to affect the current query through influence-aware text localization. QuISE then replaces these regions with two semantically distinct replacement texts that are irrelevant to both the query and the image. The final answer is determined by answer consistency across the edited images. Extensive experiments on three typographic-attack benchmarks, four attack settings, and four VLMs show that QuISE consistently improves defended accuracy. QuISE achieves a recovery rate of 67.9-75.0% with a harm rate of 0.5-1.1%.

Related papers

• SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models [9.496555964908188]
  Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues.<n>We introduce SIGNPOST-Bench, a controlled counterfactual benchmark for evaluating text-vision conflict resolution.
  arXiv  Detail & Related papers  (2026-08-04T21:55:47Z)
• Do Images Speak Louder than Words? Investigating the Effect of Textual Misinformation in VLMs [17.56537230934894]
  Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities on Visual-Question-Answering (VQA) benchmarks.<n>We show that these models are indeed vulnerable to misleading textual prompts, often overriding clear visual evidence in favor of the conflicting text.
  arXiv  Detail & Related papers  (2026-01-27T05:04:38Z)
• T2VAttack: Adversarial Attack on Text-to-Video Diffusion Models [67.13397169618624]
  We introduce T2VAttack, a study of adversarial attacks on Text-to-Video (T2V) models from both semantic and temporal perspectives.<n>To achieve an effective and efficient attack process, we propose two adversarial attack methods: T2VAttack-S, which identifies semantically or temporally critical words in prompts and replaces them with synonyms via greedy search, and T2VAttack-I, which iteratively inserts optimized words with minimal perturbation to the prompt.
  arXiv  Detail & Related papers  (2025-12-30T03:00:46Z)
• Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models [6.1438667397171685]
  Large vision-language models (LVLMs) are vulnerable to typographic attacks.<n>Existing evaluation protocols and defenses implicitly encourage ignoring text to achieve robustness.<n>We introduce Read-or-Ignore VQA (RIO-VQA), which formalizes selective text use in visual question answering.
  arXiv  Detail & Related papers  (2025-12-10T08:34:28Z)
• Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models [57.357091028792325]
  Vision-language models (VLMs) pre-trained on web-scale data exhibit promising zero-shot generalization but often suffer from semantic misalignment.<n>We propose a novel constrained prompt enhancement (CPE) method to improve visual-textual alignment.<n>Our approach consists of two key components: Topology-Guided Synonymous Semantic Generation (TGSSG) and Category-Agnostic Discriminative Region Selection (CADRS)
  arXiv  Detail & Related papers  (2025-08-24T15:45:22Z)
• Empowering Morphing Attack Detection using Interpretable Image-Text Foundation Model [3.013675405024281]
  We present a multimodal learning approach that can provide a textual description of morphing attack detection.<n>We first show that zero-shot evaluation of the proposed framework can yield not only generalizable morphing attack detection, but also predict the most relevant text snippet.
  arXiv  Detail & Related papers  (2025-08-13T18:06:29Z)
• TrojVLM: Backdoor Attack Against Vision Language Models [50.87239635292717]
  This study introduces TrojVLM, the first exploration of backdoor attacks aimed at Vision Language Models (VLMs) TrojVLM inserts predetermined target text into output text when encountering poisoned images. A novel semantic preserving loss is proposed to ensure the semantic integrity of the original image content.
  arXiv  Detail & Related papers  (2024-09-28T04:37:09Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)
• VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models [58.21452697997078]
  We propose a novel VQAttack model, which can generate both image and text perturbations with the designed modules. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQAttack.
  arXiv  Detail & Related papers  (2024-02-16T21:17:42Z)
• Verifying the Robustness of Automatic Credibility Assessment [50.55687778699995]
  We show that meaning-preserving changes in input text can mislead the models. We also introduce BODEGA: a benchmark for testing both victim models and attack methods on misinformation detection tasks. Our experimental results show that modern large language models are often more vulnerable to attacks than previous, smaller solutions.
  arXiv  Detail & Related papers  (2023-03-14T16:11:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.