Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks

• URL: http://arxiv.org/abs/2408.05025v2
• Date: Mon, 12 Aug 2024 13:57:42 GMT
• Title: Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks
• Authors: Gianluca De Stefano, Lea Schönherr, Giancarlo Pellegrino,
• Abstract summary: Retrieval Augmented Generation (RAG) is a technique commonly used to equip models with out of distribution knowledge. This paper investigates the security of RAG systems against end-to-end indirect prompt manipulations.
• Score: 12.061098193438022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval Augmented Generation (RAG) is a technique commonly used to equip models with out of distribution knowledge. This process involves collecting, indexing, retrieving, and providing information to an LLM for generating responses. Despite its growing popularity due to its flexibility and low cost, the security implications of RAG have not been extensively studied. The data for such systems are often collected from public sources, providing an attacker a gateway for indirect prompt injections to manipulate the responses of the model. In this paper, we investigate the security of RAG systems against end-to-end indirect prompt manipulations. First, we review existing RAG framework pipelines, deriving a prototypical architecture and identifying critical parameters. We then examine prior works searching for techniques that attackers can use to perform indirect prompt manipulations. Finally, we implemented Rag 'n Roll, a framework to determine the effectiveness of attacks against end-to-end RAG applications. Our results show that existing attacks are mostly optimized to boost the ranking of malicious documents during the retrieval phase. However, a higher rank does not immediately translate into a reliable attack. Most attacks, against various configurations, settle around a 40% success rate, which could rise to 60% when considering ambiguous answers as successful attacks (those that include the expected benign one as well). Additionally, when using unoptimized documents, attackers deploying two of them (or more) for a target query can achieve similar results as those using optimized ones. Finally, exploration of the configuration space of a RAG showed limited impact in thwarting the attacks, where the most successful combination severely undermines functionality.

Related papers

• Data Extraction Attacks in Retrieval-Augmented Generation via Backdoors [15.861833242429228]
  We investigate data extraction attacks targeting the knowledge databases of Retrieval-Augmented Generation (RAG) systems. To reveal the vulnerability, we propose to backdoor RAG, where a small portion of poisoned data is injected during the fine-tuning phase to create a backdoor within the LLM.
  arXiv  Detail & Related papers  (2024-11-03T22:27:40Z)
• HijackRAG: Hijacking Attacks against Retrieval-Augmented Large Language Models [18.301965456681764]
  We reveal a novel vulnerability, the retrieval prompt hijack attack (HijackRAG) HijackRAG enables attackers to manipulate the retrieval mechanisms of RAG systems by injecting malicious texts into the knowledge database. We propose both black-box and white-box attack strategies tailored to different levels of the attacker's knowledge.
  arXiv  Detail & Related papers  (2024-10-30T09:15:51Z)
• AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuning [93.77763753231338]
  Adversarial Contrastive Prompt Tuning (ACPT) is proposed to fine-tune the CLIP image encoder to extract similar embeddings for any two intermediate adversarial queries. We show that ACPT can detect 7 state-of-the-art query-based attacks with $>99%$ detection rate within 5 shots. We also show that ACPT is robust to 3 types of adaptive attacks.
  arXiv  Detail & Related papers  (2024-08-04T09:53:50Z)
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases [73.04652687616286]
  We propose AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance.
  arXiv  Detail & Related papers  (2024-07-17T17:59:47Z)
• BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models [18.107026036897132]
  Large Language Models (LLMs) are constrained by outdated information and a tendency to generate incorrect data. Retrieval-Augmented Generation (RAG) addresses these limitations by combining the strengths of retrieval-based methods and generative models. RAG introduces a new attack surface for LLMs, particularly because RAG databases are often sourced from public data, such as the web.
  arXiv  Detail & Related papers  (2024-06-03T02:25:33Z)
• Query Provenance Analysis: Efficient and Robust Defense against Query-based Black-box Attacks [11.32992178606254]
  We propose a novel approach, Query Provenance Analysis (QPA), for more robust and efficient Stateful Defense Models (SDMs) QPA encapsulates the historical relationships among queries as the sequence feature to capture the fundamental difference between benign and adversarial query sequences. We evaluate QPA compared with two baselines, BlackLight and PIHA, on four widely used datasets with six query-based black-box attack algorithms.
  arXiv  Detail & Related papers  (2024-05-31T06:56:54Z)
• Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generation [0.9217021281095907]
  We introduce an efficient and easy-to-use method for conducting a Membership Inference Attack (MIA) against RAG systems. We demonstrate the effectiveness of our attack using two benchmark datasets and multiple generative models. Our findings highlight the importance of implementing security countermeasures in deployed RAG systems.
  arXiv  Detail & Related papers  (2024-05-30T19:46:36Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• Certifiably Robust RAG against Retrieval Corruption [58.677292678310934]
  Retrieval-augmented generation (RAG) has been shown vulnerable to retrieval corruption attacks. In this paper, we propose RobustRAG as the first defense framework against retrieval corruption attacks.
  arXiv  Detail & Related papers  (2024-05-24T13:44:25Z)
• PRAT: PRofiling Adversarial aTtacks [52.693011665938734]
  We introduce a novel problem of PRofiling Adversarial aTtacks (PRAT) Given an adversarial example, the objective of PRAT is to identify the attack used to generate it. We use AID to devise a novel framework for the PRAT objective.
  arXiv  Detail & Related papers  (2023-09-20T07:42:51Z)
• Generalizable Black-Box Adversarial Attack with Meta Learning [54.196613395045595]
  In black-box adversarial attack, the target model's parameters are unknown, and the attacker aims to find a successful perturbation based on query feedback under a query budget. We propose to utilize the feedback information across historical attacks, dubbed example-level adversarial transferability. The proposed framework with the two types of adversarial transferability can be naturally combined with any off-the-shelf query-based attack methods to boost their performance.
  arXiv  Detail & Related papers  (2023-01-01T07:24:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.