Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning

• URL: http://arxiv.org/abs/2402.08416v1
• Date: Tue, 13 Feb 2024 12:40:39 GMT
• Title: Pandora: Jailbreak GPTs by Retrieval Augmented Generation Poisoning
• Authors: Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, Yang Liu,
• Abstract summary: This study investigates indirect jailbreak attacks on Large Language Models(LLMs) We introduce a novel attack vector named Retrieval Augmented Generation Poisoning. Pandora exploits the synergy between LLMs and RAG through prompt manipulation to generate unexpected responses.
• Score: 19.45092401994873
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models~(LLMs) have gained immense popularity and are being increasingly applied in various domains. Consequently, ensuring the security of these models is of paramount importance. Jailbreak attacks, which manipulate LLMs to generate malicious content, are recognized as a significant vulnerability. While existing research has predominantly focused on direct jailbreak attacks on LLMs, there has been limited exploration of indirect methods. The integration of various plugins into LLMs, notably Retrieval Augmented Generation~(RAG), which enables LLMs to incorporate external knowledge bases into their response generation such as GPTs, introduces new avenues for indirect jailbreak attacks. To fill this gap, we investigate indirect jailbreak attacks on LLMs, particularly GPTs, introducing a novel attack vector named Retrieval Augmented Generation Poisoning. This method, Pandora, exploits the synergy between LLMs and RAG through prompt manipulation to generate unexpected responses. Pandora uses maliciously crafted content to influence the RAG process, effectively initiating jailbreak attacks. Our preliminary tests show that Pandora successfully conducts jailbreak attacks in four different scenarios, achieving higher success rates than direct attacks, with 64.3\% for GPT-3.5 and 34.8\% for GPT-4.

Related papers

• SQL Injection Jailbreak: A Structural Disaster of Large Language Models [71.55108680517422]
  We introduce a novel jailbreak method, which targets the external properties of LLMs. By injecting jailbreak information into user prompts, SIJ successfully induces the model to output harmful content. We propose a simple defense method called Self-Reminder-Key to counter SIJ.
  arXiv  Detail & Related papers  (2024-11-03T13:36:34Z)
• Jigsaw Puzzles: Splitting Harmful Questions to Jailbreak Large Language Models [50.89022445197919]
  Large language models (LLMs) have exhibited outstanding performance in engaging with humans. LLMs are vulnerable to jailbreak attacks, leading to the generation of harmful responses. We propose Jigsaw Puzzles (JSP), a straightforward yet effective multi-turn jailbreak strategy against the advanced LLMs.
  arXiv  Detail & Related papers  (2024-10-15T10:07:15Z)
• EnJa: Ensemble Jailbreak on Large Language Models [69.13666224876408]
  Large Language Models (LLMs) are increasingly being deployed in safety-critical applications. LLMs can still be jailbroken by carefully crafted malicious prompts, producing content that violates policy regulations. We propose a novel EnJa attack to hide harmful instructions using prompt-level jailbreak, boost the attack success rate using a gradient-based attack, and connect the two types of jailbreak attacks via a template-based connector.
  arXiv  Detail & Related papers  (2024-08-07T07:46:08Z)
• SoP: Unlock the Power of Social Facilitation for Automatic Jailbreak Attack [16.3259723257638]
  We introduce SoP, a framework to design jailbreak prompts automatically. We show that SoP achieves attack success rates of 88% and 60% in bypassing the safety alignment of GPT-3.5-1106 and GPT-4.
  arXiv  Detail & Related papers  (2024-07-02T02:58:29Z)
• Virtual Context: Enhancing Jailbreak Attacks with Special Token Injection [54.05862550647966]
  This paper introduces Virtual Context, which leverages special tokens, previously overlooked in LLM security, to improve jailbreak attacks. Comprehensive evaluations show that Virtual Context-assisted jailbreak attacks can improve the success rates of four widely used jailbreak methods by approximately 40%.
  arXiv  Detail & Related papers  (2024-06-28T11:35:54Z)
• LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper [16.078682415975337]
  Jailbreaking is an emerging adversarial attack that bypasses the safety alignment deployed in off-the-shelf large language models (LLMs) This paper proposes a lightweight yet practical defense called SELFDEFEND. It can defend against all existing jailbreak attacks with minimal delay for jailbreak prompts and negligible delay for normal user prompts.
  arXiv  Detail & Related papers  (2024-02-24T05:34:43Z)
• All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks [0.0]
  This study introduces a straightforward black-box method for efficiently crafting jailbreak prompts. Our technique iteratively transforms harmful prompts into benign expressions directly utilizing the target LLM. Our method consistently achieved an attack success rate exceeding 80% within an average of five iterations for forbidden questions.
  arXiv  Detail & Related papers  (2024-01-18T08:36:54Z)
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically [34.36053833900958]
  We present Tree of Attacks with Pruning (TAP), an automated method for generating jailbreaks. TAP generates prompts that jailbreak state-of-the-art LLMs for more than 80% of the prompts. TAP is also capable of jailbreaking LLMs protected by state-of-the-art guardrails, e.g., LlamaGuard.
  arXiv  Detail & Related papers  (2023-12-04T18:49:23Z)
• Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts [64.60375604495883]
  We discover a system prompt leakage vulnerability in GPT-4V. By employing GPT-4 as a red teaming tool against itself, we aim to search for potential jailbreak prompts leveraging stolen system prompts. We also evaluate the effect of modifying system prompts to defend against jailbreaking attacks.
  arXiv  Detail & Related papers  (2023-11-15T17:17:39Z)
• Jailbreaking Black Box Large Language Models in Twenty Queries [97.29563503097995]
  Large language models (LLMs) are vulnerable to adversarial jailbreaks. We propose an algorithm that generates semantic jailbreaks with only black-box access to an LLM.
  arXiv  Detail & Related papers  (2023-10-12T15:38:28Z)
• Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks [12.540530764250812]
  We propose a formalism and a taxonomy of known (and possible) jailbreaks. We release a dataset of model outputs across 3700 jailbreak prompts over 4 tasks.
  arXiv  Detail & Related papers  (2023-05-24T09:57:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.