Towards Safer RAG: Only Agents Capable of System 2 Thinking may Access Untrusted Documents

• URL: http://arxiv.org/abs/2608.17153v1
• Date: Mon, 17 Aug 2026 21:37:20 GMT
• Title: Towards Safer RAG: Only Agents Capable of System 2 Thinking may Access Untrusted Documents
• Abstract summary: Retrieval-Augmented Generation (RAG) has significantly enhanced the performance of large language models.<n>RAG is vulnerable to knowledge-poisoning attacks in which misinformation in retrieved documents can influence the model's final outputs.<n>We propose a refined security principle: only agents capable of deliberative System 2 reasoning may access untrusted documents.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) has significantly enhanced the performance of large language models (LLMs), yet these systems remain vulnerable to knowledge-poisoning attacks, in which misinformation in retrieved documents can influence the model's final outputs. Notably, an LLM may correctly detect that a document contains incorrect information while nevertheless being influenced by it. Prior work has addressed this vulnerability through the Cordon Principle, which prevents models responsible for final answer synthesis from directly accessing raw evidence. Although effective, this strict isolation can introduce substantial computational overhead. In this work, we propose a refined security principle: only agents capable of deliberative System 2 reasoning may access untrusted documents. To evaluate this principle, we introduce novel metrics that quantify the discrepancy between misinformation detection and downstream influence. We then empirically compare state-of-the-art reasoning language models with standard language models across these metrics. Our results show that reasoning-capable models are substantially more robust to corrupted evidence, without requiring the strict isolation imposed by the Cordon Principle. These findings provide empirical support for our refined principle and suggest a more practical foundation for secure RAG system design.

Related papers

• VerTox: Verifiable Reward-Guided Corpus Poisoning Against Neural Ranking Models [20.029939692534292]
  This work investigates the vulnerability of neural ranking models to corpus poisoning attacks.<n>An adversary injects a small number of maliciously crafted documents into the corpus to distort ranking behavior.<n>We propose VerTox, the first framework to formulate corpus poisoning as a verifiable reward-guided reinforcement learning problem.
  arXiv  Detail & Related papers  (2026-09-01T14:43:11Z)
• Cordon-MAS: Defending RAG against Knowledge Poisoning via Information-Flow Control [22.814552066611597]
  Existing defenses assume that detecting poisoned evidence prevents harm.<n>We show this assumption is incorrect: models exhibit a monitoring-control gap.<n>We introduce the Cordon Principle -- no agent capable of final synthesis may access untrusted natural-language evidence.
  arXiv  Detail & Related papers  (2026-05-26T09:27:19Z)
• Trust but Verify: Introducing DAVinCI -- A Framework for Dual Attribution and Verification in Claim Inference for Language Models [53.279391576560755]
  Large Language Models (LLMs) have demonstrated remarkable fluency and versatility across a wide range of NLP tasks, yet they remain prone to factual inaccuracies and limitation.<n>This poses significant risks in high-stakes domains such as healthcare, law, and scientific communication, where trust and verifiability are paramount.<n>We introduce DAVinCI - a Dual Attribution and Verification framework designed to enhance the factual reliability and interpretability of LLM outputs.
  arXiv  Detail & Related papers  (2026-04-23T01:37:50Z)
• AdversarialCoT: Single-Document Retrieval Poisoning for LLM Reasoning [115.3243260783674]
  Retrieval-augmented generation (RAG) enhances large language model (LLM) reasoning by retrieving external documents.<n>We study knowledge-base poisoning attacks in RAG, where an attacker injects malicious content into the retrieval corpus.<n>We propose AdversarialCoT, a query-specific attack that poisons only a single document in the corpus.
  arXiv  Detail & Related papers  (2026-04-14T02:10:23Z)
• Verifiable Reasoning for LLM-based Generative Recommendation [106.7765000777685]
  Reasoning in Large Language Models (LLMs) has recently shown strong potential in enhancing generative recommendation.<n>We propose a novel textbftextitreason-verify-recommend paradigm, which interleaves reasoning with verification to provide reliable feedback.<n>Experiments on four real-world datasets demonstrate that VRec substantially enhances recommendation effectiveness and scalability without compromising efficiency.
  arXiv  Detail & Related papers  (2026-03-08T16:55:45Z)
• Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities [32.76303717104482]
  We introduce DeR2, a controlled deep-research sandbox that isolates document-grounded reasoning.<n>DeR2 decouples evidence access from reasoning via four regimes--Instruction-only, Concepts, Related-only, and Full-set.<n> Experiments across a diverse set of state-of-the-art foundation models reveal substantial variation and significant headroom.
  arXiv  Detail & Related papers  (2026-01-29T16:26:19Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Quantifying Document Impact in RAG-LLMs [9.10734114158633]
  We introduce the Influence Score (IS), a novel metric based on Partial Information Decomposition that measures the impact of each retrieved document on the generated response.<n>First, a poison attack simulation across three datasets demonstrates that IS correctly identifies the malicious document as the most influential in $86%$ of cases.<n>Second, an ablation study shows that a response generated using only the top-ranked documents is consistently judged more similar to the original response than one generated from the remaining documents.
  arXiv  Detail & Related papers  (2025-10-27T00:47:13Z)
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search [69.60882125603133]
  We present ReliabilityRAG, a framework for adversarial robustness that explicitly leverages reliability information of retrieved documents.<n>Our work is a significant step towards more effective, provably robust defenses against retrieved corpus corruption in RAG.
  arXiv  Detail & Related papers  (2025-09-27T22:36:42Z)
• Chain-of-Thought Poisoning Attacks against R1-based Retrieval-Augmented Generation Systems [39.05753852489526]
  Existing adversarial attack methods typically exploit knowledge base poisoning to probe the vulnerabilities of RAG systems.<n>This paper uses reasoning process templates from R1-based RAG systems to wrap erroneous knowledge into adversarial documents, and injects them into the knowledge base to attack RAG systems.<n>The key idea of our approach is that adversarial documents, by simulating the chain-of-thought patterns aligned with the model's training signals, may be misinterpreted by the model as authentic historical reasoning processes.
  arXiv  Detail & Related papers  (2025-05-22T08:22:46Z)
• TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generation [31.231916859341865]
  TrustRAG is a framework that systematically filters malicious and irrelevant content before it is retrieved for generation.<n>TrustRAG delivers substantial improvements in retrieval accuracy, efficiency, and attack resistance.
  arXiv  Detail & Related papers  (2025-01-01T15:57:34Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.