Towards Unsupervised Adversarial Document Detection in Retrieval Augmented Generation Systems

• URL: http://arxiv.org/abs/2603.17176v1
• Date: Tue, 17 Mar 2026 22:09:37 GMT
• Title: Towards Unsupervised Adversarial Document Detection in Retrieval Augmented Generation Systems
• Abstract summary: Retrieval augmented generation systems have become an integral part of everyday life.<n>Manipulating the context documents is a way to persist attacks and make them affect all users.<n>Manipulating the target prompt, which the attacker wants to manipulate, is not required for a successful detection.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval augmented generation systems have become an integral part of everyday life. Whether in internet search engines, email systems, or service chatbots, these systems are based on context retrieval and answer generation with large language models. With their spread, also the security vulnerabilities increase. Attackers become increasingly focused on these systems and various hacking approaches are developed. Manipulating the context documents is a way to persist attacks and make them affect all users. Therefore, detecting compromised, adversarial context documents early is crucial for security. While supervised approaches require a large amount of labeled adversarial contexts, we propose an unsupervised approach, being able to detect also zero day attacks. We conduct a preliminary study to show appropriate indicators for adversarial contexts. For that purpose generator activations, output embeddings, and an entropy-based uncertainty measure turn out as suitable, complementary quantities. With an elementary statistical outlier detection, we propose and compare their detection abilities. Furthermore, we show that the target prompt, which the attacker wants to manipulate, is not required for a successful detection. Moreover, our results indicate that a simple context summary generation might even be superior in finding manipulated contexts.

Related papers

• Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs [65.6660735371212]
  We present textbftextscJustAsk, a framework that autonomously discovers effective extraction strategies through interaction alone.<n>It formulates extraction as an online exploration problem, using Upper Confidence Bound--based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration.<n>Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.
  arXiv  Detail & Related papers  (2026-01-29T03:53:25Z)
• CAMOUFLAGE: Exploiting Misinformation Detection Systems Through LLM-driven Adversarial Claim Transformation [4.02943411607022]
  Existing black-box text-based adversarial attacks are ill-suited for evidence-based misinformation detection systems.<n>We present CAMOUFLAGE, an iterative, LLM-driven approach that employs a two-agent system to create adversarial claim rewritings.<n>We evaluate CAMOUFLAGE on four systems, including two recent academic systems and two real-world APIs, with an average attack success rate of 46.92%.
  arXiv  Detail & Related papers  (2025-05-03T19:14:24Z)
• Illusions of Relevance: Using Content Injection Attacks to Deceive Retrievers, Rerankers, and LLM Judges [52.96987928118327]
  We find that embedding models for retrieval, rerankers, and large language model (LLM) relevance judges are vulnerable to content injection attacks.<n>We identify two primary threats: (1) inserting unrelated or harmful content within passages that still appear deceptively "relevant", and (2) inserting entire queries or key query terms into passages to boost their perceived relevance.<n>Our study systematically examines the factors that influence an attack's success, such as the placement of injected content and the balance between relevant and non-relevant material.
  arXiv  Detail & Related papers  (2025-01-30T18:02:15Z)
• Humanizing Machine-Generated Content: Evading AI-Text Detection through Adversarial Attack [24.954755569786396]
  We propose a framework for a broader class of adversarial attacks, designed to perform minor perturbations in machine-generated content to evade detection. We consider two attack settings: white-box and black-box, and employ adversarial learning in dynamic scenarios to assess the potential enhancement of the current detection model's robustness. The empirical results reveal that the current detection models can be compromised in as little as 10 seconds, leading to the misclassification of machine-generated text as human-written content.
  arXiv  Detail & Related papers  (2024-04-02T12:49:22Z)
• Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information [67.78183175605761]
  Large Language Models are susceptible to adversarial prompt attacks. This vulnerability underscores a significant concern regarding the robustness and reliability of LLMs. We introduce a novel approach to detecting adversarial prompts at a token level.
  arXiv  Detail & Related papers  (2023-11-20T03:17:21Z)
• Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection [12.244543468021938]
  This paper introduces two types of detection tasks for adversarial documents. A benchmark dataset is established to facilitate the investigation of adversarial ranking defense. A comprehensive investigation of the performance of several detection baselines is conducted.
  arXiv  Detail & Related papers  (2023-07-31T16:31:24Z)
• Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attacks [76.35478518372692]
  We introduce epsilon-illusory, a novel form of adversarial attack on sequential decision-makers. Compared to existing attacks, we empirically find epsilon-illusory to be significantly harder to detect with automated methods. Our findings suggest the need for better anomaly detectors, as well as effective hardware- and system-level defenses.
  arXiv  Detail & Related papers  (2022-07-20T19:49:09Z)
• Zero-Query Transfer Attacks on Context-Aware Object Detectors [95.18656036716972]
  Adversarial attacks perturb images such that a deep neural network produces incorrect classification results. A promising approach to defend against adversarial attacks on natural multi-object scenes is to impose a context-consistency check. We present the first approach for generating context-consistent adversarial attacks that can evade the context-consistency check.
  arXiv  Detail & Related papers  (2022-03-29T04:33:06Z)
• No Need to Know Physics: Resilience of Process-based Model-free Anomaly Detection for Industrial Control Systems [95.54151664013011]
  We present a novel framework to generate adversarial spoofing signals that violate physical properties of the system. We analyze four anomaly detectors published at top security conferences.
  arXiv  Detail & Related papers  (2020-12-07T11:02:44Z)
• Universal Adversarial Attacks with Natural Triggers for Text Classification [30.74579821832117]
  We develop adversarial attacks that appear closer to natural English phrases and yet confuse classification systems. Our attacks effectively reduce model accuracy on classification tasks while being less identifiable than prior models.
  arXiv  Detail & Related papers  (2020-05-01T01:58:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.