RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts

• URL: http://arxiv.org/abs/2510.05310v1
• Date: Mon, 06 Oct 2025 19:20:43 GMT
• Title: RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contexts
• Authors: Yining She, Daniel W. Peterson, Marianne Menglin Liu, Vikas Upadhyay, Mohammad Hossein Chaghazardi, Eunsuk Kang, Dan Roth,
• Abstract summary: External LLM-based guardrail models have emerged as a popular solution to screen unsafe inputs and outputs.<n>We investigated how robust LLM-based guardrails are against additional information embedded in the context.
• Score: 39.58550043591753
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the increasing adoption of large language models (LLMs), ensuring the safety of LLM systems has become a pressing concern. External LLM-based guardrail models have emerged as a popular solution to screen unsafe inputs and outputs, but they are themselves fine-tuned or prompt-engineered LLMs that are vulnerable to data distribution shifts. In this paper, taking Retrieval Augmentation Generation (RAG) as a case study, we investigated how robust LLM-based guardrails are against additional information embedded in the context. Through a systematic evaluation of 3 Llama Guards and 2 GPT-oss models, we confirmed that inserting benign documents into the guardrail context alters the judgments of input and output guardrails in around 11% and 8% of cases, making them unreliable. We separately analyzed the effect of each component in the augmented context: retrieved documents, user query, and LLM-generated response. The two mitigation methods we tested only bring minor improvements. These results expose a context-robustness gap in current guardrails and motivate training and evaluation protocols that are robust to retrieval and query composition.

Related papers

• Taxonomy-Adaptive Moderation Model with Robust Guardrails for Large Language Models [3.710103086278309]
  Large Language Models (LLMs) are typically aligned for safety during the post-training phase.<n>They may still generate inappropriate outputs that could potentially pose risks to users.<n>This challenge underscores the need for robust safeguards that operate across both model inputs and outputs.
  arXiv  Detail & Related papers  (2025-12-05T00:43:55Z)
• Black-Box Guardrail Reverse-engineering Attack [12.937652779951156]
  We present the first study of black-box LLM guardrail reverse-engineering attacks.<n>We propose Guardrail Reverse-engineering Attack (GRA), a reinforcement learning-based framework.<n>GRA achieves a rule matching rate exceeding 0.92 while requiring less than $85 in API costs.
  arXiv  Detail & Related papers  (2025-11-06T09:24:49Z)
• LLM Embedding-based Attribution (LEA): Quantifying Source Contributions to Generative Model's Response for Vulnerability Analysis [1.3543506826034255]
  Large Language Models (LLMs) are increasingly used for cybersecurity threat analysis, but their deployment in security-sensitive environments raises trust and safety concerns.<n>This work proposes Embedding Attribution (LEA) to analyze the generated responses for vulnerability exploitation analysis.<n>Our results demonstrate LEA's ability to detect clear distinctions between non-retrieval, generic-retrieval, and valid-retrieval scenarios with over 95% accuracy on larger models.
  arXiv  Detail & Related papers  (2025-06-12T21:20:10Z)
• Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs [54.90315421117162]
  We propose a novel poisoning method via completely harmless data.<n>Inspired by the causal reasoning in auto-regressive LLMs, we aim to establish robust associations between triggers and an affirmative response prefix.<n>We observe an interesting resistance phenomenon where the LLM initially appears to agree but subsequently refuses to answer.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generation [31.231916859341865]
  TrustRAG is a framework that systematically filters malicious and irrelevant content before it is retrieved for generation.<n>TrustRAG delivers substantial improvements in retrieval accuracy, efficiency, and attack resistance.
  arXiv  Detail & Related papers  (2025-01-01T15:57:34Z)
• Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks [45.07581174558107]
  Retrieval-Augmented Generation (RAG) systems have emerged as a promising solution to mitigate hallucinations.<n>RAG systems are vulnerable to adversarial poisoning attacks, where malicious passages injected into the retrieval corpus can mislead models into producing factually incorrect outputs.<n>We present a rigorously controlled empirical study of how RAG systems behave under such attacks and how their robustness can be improved.
  arXiv  Detail & Related papers  (2024-12-21T17:31:52Z)
• Invar-RAG: Invariant LLM-aligned Retrieval for Better Generation [43.630437906898635]
  We propose a novel two-stage fine-tuning architecture called Invar-RAG. In the retrieval stage, an LLM-based retriever is constructed by integrating LoRA-based representation learning. In the generation stage, a refined fine-tuning method is employed to improve LLM accuracy in generating answers based on retrieved information.
  arXiv  Detail & Related papers  (2024-11-11T14:25:37Z)
• ShieldGemma: Generative AI Content Moderation Based on Gemma [49.91147965876678]
  ShieldGemma is a suite of safety content moderation models built upon Gemma2. Models provide robust, state-of-the-art predictions of safety risks across key harm types.
  arXiv  Detail & Related papers  (2024-07-31T17:48:14Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)
• Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs [59.596335292426105]
  This paper collects the first open-source dataset to evaluate safeguards in large language models. We train several BERT-like classifiers to achieve results comparable with GPT-4 on automatic safety evaluation.
  arXiv  Detail & Related papers  (2023-08-25T14:02:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.