ViTeGate: Visual-Textual Triggered Knowledge Poisoning for Vision-Language Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2609.14685v1
• Date: Sun, 13 Sep 2026 17:39:37 GMT
• Title: ViTeGate: Visual-Textual Triggered Knowledge Poisoning for Vision-Language Retrieval-Augmented Generation
• Abstract summary: ViTeGate is a visual-textual triggered knowledge poisoning attack for Vision-Language Retrieval-Augmented Generation (VLRAG) systems.<n>By coordinating retrieval and generation, ViTeGate reduces poison exposure when the visual trigger is absent and preserves normal responses when the textual trigger is absent.<n>On InfoSeek, ViTeGate achieves an attack success rate of up to 0.98 while maintaining a clean answer accuracy of up to 0.93.
• Score: 10.286163249620545
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern Vision-Language Retrieval-Augmented Generation (VLRAG) systems augment Large Vision-Language Models (LVLMs) with retrieved visual and textual evidence, enabling responses grounded in external knowledge. However, the retrieval pipeline also creates an attack surface: adversaries can inject poisoned image-text pairs into the knowledge corpus to influence model outputs. Existing knowledge poisoning attacks are typically always-on, allowing poisoned evidence to affect generation whenever it is retrieved. This lack of precise activation control makes it difficult to confine malicious behavior to intended inputs, reducing both attack stealth and effectiveness. In this paper, we propose ViTeGate, a visual-textual triggered knowledge poisoning attack for VLRAG systems. ViTeGate uses a visual trigger to conditionally promote poisoned evidence into retrieval results and a textual trigger to induce an attacker-specified response from the retrieved evidence. By coordinating retrieval and generation, ViTeGate reduces poison exposure when the visual trigger is absent and preserves normal responses when the textual trigger is absent. The two-trigger design enables selective attack activation and reduces unintended single-trigger activation. Experiments across multiple query datasets, retrievers, and LVLMs validate the effectiveness of ViTeGate. On InfoSeek, ViTeGate achieves an attack success rate of up to 0.98 while maintaining a clean answer accuracy of up to 0.93.

Related papers

• When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse [34.49107596901488]
  RAGs are increasingly susceptible to poisoning attacks, in which adversarial documents are injected to manipulate generator outputs.<n>Previous methods rely on output-side signals such as perplexity and consistency checks to detect such attacks.<n>We propose textttD-SCAN, a lightweight detection framework that monitors attention dynamics to identify attacked generations.
  arXiv  Detail & Related papers  (2026-08-07T08:23:26Z)
• Beyond Attack Success Rate: Examining Trigger Leakage in Vision-Language Agentic Systems [36.30772806498619]
  Vision-Language Agentic Systems (VLAS) connect visual perception to planning, tool use, and physical actions.<n>Backdoor-type triggers can propagate through both decision pipelines and their connected interfaces.
  arXiv  Detail & Related papers  (2026-06-10T18:33:52Z)
• Reasoning-Style Poisoning of LLM Agents via Stealthy Style Transfer: Process-Level Attacks and Runtime Monitoring in RSV Space [4.699272847316498]
  Reasoning-Style Poisoning (RSP) manipulates how agents process information rather than what they process.<n>Generative Style Injection (GSI) rewrites retrieved documents into pathological tones.<n>RSP-M is a lightweight runtime monitor that calculates RSV metrics in real-time and triggers alerts when values exceed safety thresholds.
  arXiv  Detail & Related papers  (2025-12-16T14:34:10Z)
• TokenSwap: Backdoor Attack on the Compositional Understanding of Large Vision-Language Models [57.32952956674526]
  We introduce TokenSwap, a more evasive and stealthy backdoor attack on large vision-language models (LVLMs)<n>Instead of enforcing a fixed targeted content, TokenSwap subtly disrupts the understanding of object relationships in text.<n> TokenSwap achieves high attack success rates while maintaining superior evasiveness and stealthiness.
  arXiv  Detail & Related papers  (2025-09-29T10:19:22Z)
• Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents [54.35629963816521]
  This work introduces VIBMA, the first clean-text backdoor attack targeting VLM-based mobile agents.<n>The attack injects malicious behaviors into the model by modifying only the visual input.<n>We show that our attack achieves high success rates while preserving clean-task behavior.
  arXiv  Detail & Related papers  (2025-06-16T08:09:32Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• Practical Poisoning Attacks against Retrieval-Augmented Generation [9.320227105592917]
  Large language models (LLMs) have demonstrated impressive natural language processing abilities but face challenges such as hallucination and outdated knowledge.<n>Retrieval-Augmented Generation (RAG) has emerged as a state-of-the-art approach to mitigate these issues.<n>We propose CorruptRAG, a practical poisoning attack against RAG systems in which the attacker injects only a single poisoned text.
  arXiv  Detail & Related papers  (2025-04-04T21:49:42Z)
• Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation [71.32665836294103]
  Multimodal retrieval-augmented generation (RAG) enhances the visual reasoning capability of vision-language models (VLMs)<n>In this work, we introduce textitPoisoned-MRAG, the first knowledge poisoning attack on multimodal RAG systems.
  arXiv  Detail & Related papers  (2025-03-08T15:46:38Z)
• Rethinking Graph Backdoor Attacks: A Distribution-Preserving Perspective [33.35835060102069]
  Graph Neural Networks (GNNs) have shown remarkable performance in various tasks. Backdoor attack poisons the graph by attaching backdoor triggers and the target class label to a set of nodes in the training graph. In this paper, we study a novel problem of unnoticeable graph backdoor attacks with in-distribution (ID) triggers.
  arXiv  Detail & Related papers  (2024-05-17T13:09:39Z)
• PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models [45.409248316497674]
  Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG.
  arXiv  Detail & Related papers  (2024-02-12T18:28:36Z)
• Vision-LLMs Can Fool Themselves with Self-Generated Typographic Attacks [58.10730906004818]
  Typographic attacks, adding misleading text to images, can deceive vision-language models (LVLMs)<n>Our experiments show these attacks significantly reduce classification performance by up to 60%.
  arXiv  Detail & Related papers  (2024-02-01T14:41:20Z)
• Exploring Model Dynamics for Accumulative Poisoning Discovery [62.08553134316483]
  We propose a novel information measure, namely, Memorization Discrepancy, to explore the defense via the model-level information. By implicitly transferring the changes in the data manipulation to that in the model outputs, Memorization Discrepancy can discover the imperceptible poison samples. We thoroughly explore its properties and propose Discrepancy-aware Sample Correction (DSC) to defend against accumulative poisoning attacks.
  arXiv  Detail & Related papers  (2023-06-06T14:45:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.