SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model

• URL: http://arxiv.org/abs/2501.18636v2
• Date: Sun, 23 Feb 2025 10:46:28 GMT
• Title: SafeRAG: Benchmarking Security in Retrieval-Augmented Generation of Large Language Model
• Authors: Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Jason Zhaoxin Fan, Bo Tang, Shichao Song, Mengwei Wang, Jiawei Yang,
• Abstract summary: We introduce a benchmark named SafeRAG designed to evaluate the RAG security.<n>First, we classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service.<n>We then utilize the SafeRAG dataset to simulate various attack scenarios that RAG may encounter.
• Score: 17.046058202577985
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The indexing-retrieval-generation paradigm of retrieval-augmented generation (RAG) has been highly successful in solving knowledge-intensive tasks by integrating external knowledge into large language models (LLMs). However, the incorporation of external and unverified knowledge increases the vulnerability of LLMs because attackers can perform attack tasks by manipulating knowledge. In this paper, we introduce a benchmark named SafeRAG designed to evaluate the RAG security. First, we classify attack tasks into silver noise, inter-context conflict, soft ad, and white Denial-of-Service. Next, we construct RAG security evaluation dataset (i.e., SafeRAG dataset) primarily manually for each task. We then utilize the SafeRAG dataset to simulate various attack scenarios that RAG may encounter. Experiments conducted on 14 representative RAG components demonstrate that RAG exhibits significant vulnerability to all attack tasks and even the most apparent attack task can easily bypass existing retrievers, filters, or advanced LLMs, resulting in the degradation of RAG service quality. Code is available at: https://github.com/IAAR-Shanghai/SafeRAG.

Related papers

• MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAG [65.0423152595537]
  We propose MES-RAG, which enhances entity-specific query handling and provides accurate, secure, and consistent responses. MES-RAG introduces proactive security measures that ensure system integrity by applying protections prior to data access. Experimental results demonstrate that MES-RAG significantly improves both accuracy and recall, highlighting its effectiveness in advancing the security and utility of question-answering.
  arXiv  Detail & Related papers  (2025-03-17T08:09:42Z)
• MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks [109.53357276796655]
  Multimodal large language models (MLLMs) equipped with Retrieval Augmented Generation (RAG) RAG enhances MLLMs by grounding responses in query-relevant external knowledge. This reliance poses a critical yet underexplored safety risk: knowledge poisoning attacks. We propose MM-PoisonRAG, a novel knowledge poisoning attack framework with two attack strategies.
  arXiv  Detail & Related papers  (2025-02-25T04:23:59Z)
• HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models [18.301965456681764]
  We reveal a novel vulnerability, the retrieval prompt hijack attack (HijackRAG) HijackRAG enables attackers to manipulate the retrieval mechanisms of RAG systems by injecting malicious texts into the knowledge database. We propose both black-box and white-box attack strategies tailored to different levels of the attacker's knowledge.
  arXiv  Detail & Related papers  (2024-10-30T09:15:51Z)
• Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks [12.061098193438022]
  Retrieval Augmented Generation (RAG) is a technique commonly used to equip models with out of distribution knowledge. This paper investigates the security of RAG systems against end-to-end indirect prompt manipulations.
  arXiv  Detail & Related papers  (2024-08-09T12:26:05Z)
• ConfusedPilot: Confused Deputy Risks in RAG-based LLMs [2.423202571519879]
  We introduce ConfusedPilot, a class of security vulnerabilities of RAG systems that confuse Copilot and cause integrity and confidentiality violations in its responses. This study highlights the security vulnerabilities in today's RAG-based systems and proposes design guidelines to secure future RAG-based systems.
  arXiv  Detail & Related papers  (2024-08-09T05:20:05Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [69.4501863547618]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios. With a focus on factual accuracy, we propose three novel metrics Completeness, Hallucination, and Irrelevance. Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• "Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models [74.05368440735468]
  Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases.
  arXiv  Detail & Related papers  (2024-06-26T05:36:23Z)
• Certifiably Robust RAG against Retrieval Corruption [58.677292678310934]
  Retrieval-augmented generation (RAG) has been shown vulnerable to retrieval corruption attacks. In this paper, we propose RobustRAG as the first defense framework against retrieval corruption attacks.
  arXiv  Detail & Related papers  (2024-05-24T13:44:25Z)
• Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations [9.209974698634175]
  Retrieval-Augmented Generation (RAG) is a promising solution for addressing the limitations of Large Language Models (LLMs) In this work, we investigate two underexplored aspects when assessing the robustness of RAG. We introduce a novel attack method, the Genetic Attack on RAG (textitGARAG), which targets these aspects.
  arXiv  Detail & Related papers  (2024-04-22T07:49:36Z)
• ActiveRAG: Autonomously Knowledge Assimilation and Accommodation through Retrieval-Augmented Agents [49.30553350788524]
  Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to leverage external knowledge. Existing RAG models often treat LLMs as passive recipients of information. We introduce ActiveRAG, a multi-agent framework that mimics human learning behavior.
  arXiv  Detail & Related papers  (2024-02-21T06:04:53Z)
• On the Security Risks of Knowledge Graph Reasoning [71.64027889145261]
  We systematize the security threats to KGR according to the adversary's objectives, knowledge, and attack vectors. We present ROAR, a new class of attacks that instantiate a variety of such threats. We explore potential countermeasures against ROAR, including filtering of potentially poisoning knowledge and training with adversarially augmented queries.
  arXiv  Detail & Related papers  (2023-05-03T18:47:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.