A StrongREJECT for Empty Jailbreaks

• URL: http://arxiv.org/abs/2402.10260v2
• Date: Tue, 27 Aug 2024 03:32:47 GMT
• Title: A StrongREJECT for Empty Jailbreaks
• Authors: Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, Sam Toyer,
• Abstract summary: StrongREJECT is a high-quality benchmark for evaluating jailbreak performance. It scores the harmfulness of a victim model's responses to forbidden prompts. It achieves state-of-the-art agreement with human judgments of jailbreak effectiveness.
• Score: 72.8807309802266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most jailbreak papers claim the jailbreaks they propose are highly effective, often boasting near-100% attack success rates. However, it is perhaps more common than not for jailbreak developers to substantially exaggerate the effectiveness of their jailbreaks. We suggest this problem arises because jailbreak researchers lack a standard, high-quality benchmark for evaluating jailbreak performance, leaving researchers to create their own. To create a benchmark, researchers must choose a dataset of forbidden prompts to which a victim model will respond, along with an evaluation method that scores the harmfulness of the victim model's responses. We show that existing benchmarks suffer from significant shortcomings and introduce the StrongREJECT benchmark to address these issues. StrongREJECT's dataset contains prompts that victim models must answer with specific, harmful information, while its automated evaluator measures the extent to which a response gives useful information to forbidden prompts. In doing so, the StrongREJECT evaluator achieves state-of-the-art agreement with human judgments of jailbreak effectiveness. Notably, we find that existing evaluation methods significantly overstate jailbreak effectiveness compared to human judgments and the StrongREJECT evaluator. We describe a surprising and novel phenomenon that explains this discrepancy: jailbreaks bypassing a victim model's safety fine-tuning tend to reduce its capabilities. Together, our findings underscore the need for researchers to use a high-quality benchmark, such as StrongREJECT, when developing new jailbreak attacks. We release the StrongREJECT code and data at https://strong-reject.readthedocs.io/en/latest/.

Related papers

• Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring [47.40698758003993]
  We propose an improved transfer attack method that guides malicious prompt construction by locally training a mirror model of the target black-box model through benign data distillation. Our approach achieved a maximum attack success rate of 92%, or a balanced value of 80% with an average of 1.5 detectable jailbreak queries per sample against GPT-3.5 Turbo.
  arXiv  Detail & Related papers  (2024-10-28T14:48:05Z)
• EnJa: Ensemble Jailbreak on Large Language Models [69.13666224876408]
  Large Language Models (LLMs) are increasingly being deployed in safety-critical applications. LLMs can still be jailbroken by carefully crafted malicious prompts, producing content that violates policy regulations. We propose a novel EnJa attack to hide harmful instructions using prompt-level jailbreak, boost the attack success rate using a gradient-based attack, and connect the two types of jailbreak attacks via a template-based connector.
  arXiv  Detail & Related papers  (2024-08-07T07:46:08Z)
• WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models [66.34505141027624]
  We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks.
  arXiv  Detail & Related papers  (2024-06-26T17:31:22Z)
• "Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak [41.03991506580652]
  "Jailbreak" is a major safety concern of Large Language Models (LLMs) "BabyBLUE" introduces a specialized validation framework to enhance existing jailbreak benchmarks.
  arXiv  Detail & Related papers  (2024-06-17T15:51:01Z)
• JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models [21.854909839996612]
  Jailbreak attacks aim to induce Large Language Models (LLMs) to generate harmful responses for forbidden instructions. There is (surprisingly) no consensus on how to evaluate whether a jailbreak attempt is successful. JailbreakEval is a user-friendly toolkit focusing on the evaluation of jailbreak attempts.
  arXiv  Detail & Related papers  (2024-06-13T16:59:43Z)
• AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens [83.08119913279488]
  We present a systematic analysis of the dependency relationships in jailbreak attack and defense techniques. We propose three comprehensive, automated, and logical frameworks. We show that the proposed ensemble jailbreak attack and defense framework significantly outperforms existing research.
  arXiv  Detail & Related papers  (2024-06-06T07:24:41Z)
• Rethinking How to Evaluate Language Model Jailbreak [16.301224741410312]
  We propose three metrics, safeguard violation, informativeness, and relative truthfulness, to evaluate language model jailbreak. We evaluate our metrics on a benchmark dataset produced from three malicious intent datasets and three jailbreak systems.
  arXiv  Detail & Related papers  (2024-04-09T15:54:16Z)
• JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models [123.66104233291065]
  Jailbreak attacks cause large language models (LLMs) to generate harmful, unethical, or otherwise objectionable content. evaluating these attacks presents a number of challenges, which the current collection of benchmarks and evaluation techniques do not adequately address. JailbreakBench is an open-sourced benchmark with the following components.
  arXiv  Detail & Related papers  (2024-03-28T02:44:02Z)
• Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning [6.579419241184795]
  We show that moderate WANDA pruning can enhance resistance to jailbreaking attacks without fine-tuning. We introduce a dataset of 225 harmful tasks across five categories. We find that LLaMA-2 is much safer on AdvBench prompts than on our dataset when evaluated with manual jailbreak attempts.
  arXiv  Detail & Related papers  (2024-01-19T18:05:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.