Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation

• URL: http://arxiv.org/abs/2608.20756v2
• Date: Fri, 28 Aug 2026 13:48:52 GMT
• Title: Vis-Poison: Poisoning Visual Knowledge in Multimodal Retrieval-Augmented Generation
• Abstract summary: We introduce Vis-Poison, a novel visual knowledge poisoning attack.<n>It is instantiated through an automated multi-agent method that constructs visually plausible poisoned images.<n>Vis-Poison achieves an end-to-end attack success rate of 40.16% to 65.40% against 30k-entry multimodal knowledge bases.
• Score: 5.04815921134267
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While multimodal retrieval-augmented generation (RAG) systems increasingly rely on images as external knowledge sources, the introduction of poisoned visual evidence can severely compromise multimodal large language model (MLLM) generation. Unlike prior attacks that rely on altering textual metadata, we introduce Vis-Poison, a novel visual knowledge poisoning attack where the poisoned image itself is the attacker-controlled payload, without manipulating captions, summaries, metadata, or other associated text. Specifically, this attack is instantiated through an automated multi-agent method that constructs visually plausible poisoned images. To assess its impact, we evaluate Vis-Poison across two representative multimodal RAG pipelines, four embedding models, and six generation models. Empirically, Vis-Poison achieves an end-to-end attack success rate of 40.16% to 65.40% against 30k-entry multimodal knowledge bases in \emph{black-box} settings. Moreover, Vis-Poison remains effective against various MLLMs that can answer correctly from parametric knowledge alone, with an average success rate above 60%. Code and data are available at https://github.com/SWUFE-DB-Group/Vis-Poison.

Related papers

• MemVenom: Triggered Poisoning of Multimodal Memories in Web Agents [74.64265314956441]
  We propose a unified black-box attack framework that poisons graph-structured external memory with coordinated text-image evidence.<n>MemVenom achieves strong end-to-end attack success with minimal impact on benign performance, reaching up to 99.15% on GPT-5-family web agents.
  arXiv  Detail & Related papers  (2026-06-09T11:53:25Z)
• Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generation [0.8103046443444949]
  We present MM-MEPA, a multimodal poisoning attack that targets the metadata components of image-text entries while leaving the associated visual content unaltered.<n> MM-MEPA achieves an attack success rate of up to 91% consistently disrupting system behaviors across four retrievers and two multimodal generators.
  arXiv  Detail & Related papers  (2026-02-26T15:59:45Z)
• Contextual Image Attack: How Visual Context Exposes Multimodal Safety Vulnerabilities [34.64588827428617]
  We propose a new image-centric attack method, Contextual Image Attack (CIA)<n>CIA embeds harmful queries into seemingly benign visual contexts using four distinct visualization strategies.<n>Our method significantly outperforms prior work, demonstrating that the visual modality itself is a potent vector for jailbreaking advanced MLLMs.
  arXiv  Detail & Related papers  (2025-12-02T17:51:02Z)
• ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs [72.8646625127485]
  Multimodal implicit toxicity appears not only as formal statements in social platforms but also prompts that can lead to toxic dialogs.<n>Despite the success in unimodal text or image moderation, toxicity detection for multimodal content, particularly the multimodal implicit toxicity, remains underexplored.<n>To advance the detection of multimodal implicit toxicity, we build ShieldVLM, a model which identifies implicit toxicity in multimodal statements, prompts and dialogs via deliberative cross-modal reasoning.
  arXiv  Detail & Related papers  (2025-05-20T07:31:17Z)
• Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation [71.32665836294103]
  Multimodal retrieval-augmented generation (RAG) enhances the visual reasoning capability of vision-language models (VLMs)<n>In this work, we introduce textitPoisoned-MRAG, the first knowledge poisoning attack on multimodal RAG systems.
  arXiv  Detail & Related papers  (2025-03-08T15:46:38Z)
• MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks [104.50239783909063]
  Multimodal large language models with Retrieval Augmented Generation (RAG) have significantly advanced tasks such as multimodal question answering.<n>This reliance on external knowledge poses a critical yet underexplored safety risk: knowledge poisoning attacks.<n>We propose MM-PoisonRAG, the first framework to systematically design knowledge poisoning in multimodal RAG.
  arXiv  Detail & Related papers  (2025-02-25T04:23:59Z)
• Exploring Model Dynamics for Accumulative Poisoning Discovery [62.08553134316483]
  We propose a novel information measure, namely, Memorization Discrepancy, to explore the defense via the model-level information. By implicitly transferring the changes in the data manipulation to that in the model outputs, Memorization Discrepancy can discover the imperceptible poison samples. We thoroughly explore its properties and propose Discrepancy-aware Sample Correction (DSC) to defend against accumulative poisoning attacks.
  arXiv  Detail & Related papers  (2023-06-06T14:45:24Z)
• Systematic Evaluation of Backdoor Data Poisoning Attacks on Image Classifiers [6.352532169433872]
  Backdoor data poisoning attacks have been demonstrated in computer vision research as a potential safety risk for machine learning (ML) systems. Our work builds upon prior backdoor data-poisoning research for ML image classifiers. We find that poisoned models are hard to detect through performance inspection alone.
  arXiv  Detail & Related papers  (2020-04-24T02:58:22Z)
• MetaPoison: Practical General-purpose Clean-label Data Poisoning [58.13959698513719]
  Data poisoning is an emerging threat in the context of neural networks. We propose MetaPoison, a first-order method that approximates the bilevel problem via meta-learning and crafts poisons that fool neural networks. We demonstrate for the first time successful data poisoning of models trained on the black-box Google Cloud AutoML API.
  arXiv  Detail & Related papers  (2020-04-01T04:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.