WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

• URL: http://arxiv.org/abs/2406.18510v1
• Date: Wed, 26 Jun 2024 17:31:22 GMT
• Title: WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
• Authors: Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, Nouha Dziri,
• Abstract summary: We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks.
• Score: 66.34505141027624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics, and then composes multiple tactics for systematic exploration of novel jailbreaks. Compared to prior work that performed red-teaming via recruited human workers, gradient-based optimization, or iterative revision with LLMs, our work investigates jailbreaks from chatbot users who were not specifically instructed to break the system. WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks compared to state-of-the-art jailbreak methods. While many datasets exist for jailbreak evaluation, very few open-source datasets exist for jailbreak training, as safety training data has been closed even when model weights are open. With WildTeaming we create WildJailbreak, a large-scale open-source synthetic safety dataset with 262K vanilla (direct request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate exaggerated safety behaviors, WildJailbreak provides two contrastive types of queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that resemble harmful queries in form but contain no harm. As WildJailbreak considerably upgrades the quality and scale of existing safety resources, it uniquely enables us to examine the scaling effects of data and the interplay of data properties and model capabilities during safety training. Through extensive experiments, we identify the training properties that enable an ideal balance of safety behaviors: appropriate safeguarding without over-refusal, effective handling of vanilla and adversarial queries, and minimal, if any, decrease in general capabilities. All components of WildJailbeak contribute to achieving balanced safety behaviors of models.

Related papers

• SQL Injection Jailbreak: a structural disaster of large language models [71.55108680517422]
  We propose a novel jailbreak method, which utilizes the construction of input prompts by LLMs to inject jailbreak information into user prompts. Our SIJ method achieves nearly 100% attack success rates on five well-known open-source LLMs in the context of AdvBench.
  arXiv  Detail & Related papers  (2024-11-03T13:36:34Z)
• Transferable Ensemble Black-box Jailbreak Attacks on Large Language Models [0.0]
  We propose a novel black-box jailbreak attacking framework that incorporates various LLM-as-Attacker methods. Our method is designed based on three key observations from existing jailbreaking studies and practices.
  arXiv  Detail & Related papers  (2024-10-31T01:55:33Z)
• EnJa: Ensemble Jailbreak on Large Language Models [69.13666224876408]
  Large Language Models (LLMs) are increasingly being deployed in safety-critical applications. LLMs can still be jailbroken by carefully crafted malicious prompts, producing content that violates policy regulations. We propose a novel EnJa attack to hide harmful instructions using prompt-level jailbreak, boost the attack success rate using a gradient-based attack, and connect the two types of jailbreak attacks via a template-based connector.
  arXiv  Detail & Related papers  (2024-08-07T07:46:08Z)
• RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent [24.487441771427434]
  We propose a multi-agent LLM system named RedAgent to generate context-aware jailbreak prompts. Our system can jailbreak most black-box LLMs in just five queries, improving the efficiency of existing red teaming methods by two times. We have reported all found issues and communicated with OpenAI and Meta for bug fixes.
  arXiv  Detail & Related papers  (2024-07-23T17:34:36Z)
• Safe Unlearning: A Surprisingly Effective and Generalizable Solution to Defend Against Jailbreak Attacks [89.54736699767315]
  We conjecture that directly unlearn the harmful knowledge in the LLM can be a more effective way to defend against jailbreak attacks. Our solution reduced the Attack Success Rate (ASR) in Vicuna-7B from 82.6% to 7.7% on out-of-distribution (OOD) harmful questions. This significantly outperforms Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but still has an ASR of 21.9% even under the help of an additional safety system prompt.
  arXiv  Detail & Related papers  (2024-07-03T07:14:05Z)
• Virtual Context: Enhancing Jailbreak Attacks with Special Token Injection [54.05862550647966]
  This paper introduces Virtual Context, which leverages special tokens, previously overlooked in LLM security, to improve jailbreak attacks. Comprehensive evaluations show that Virtual Context-assisted jailbreak attacks can improve the success rates of four widely used jailbreak methods by approximately 40%.
  arXiv  Detail & Related papers  (2024-06-28T11:35:54Z)
• EasyJailbreak: A Unified Framework for Jailbreaking Large Language Models [53.87416566981008]
  This paper introduces EasyJailbreak, a unified framework simplifying the construction and evaluation of jailbreak attacks against Large Language Models (LLMs) It builds jailbreak attacks using four components: Selector, Mutator, Constraint, and Evaluator. Our validation across 10 distinct LLMs reveals a significant vulnerability, with an average breach probability of 60% under various jailbreaking attacks.
  arXiv  Detail & Related papers  (2024-03-18T18:39:53Z)
• Comprehensive Assessment of Jailbreak Attacks Against LLMs [28.58973312098698]
  We study 13 cutting-edge jailbreak methods from four categories, 160 questions from 16 violation categories, and six popular LLMs. Our experimental results demonstrate that the optimized jailbreak prompts consistently achieve the highest attack success rates. We discuss the trade-off between the attack performance and efficiency, as well as show that the transferability of the jailbreak prompts is still viable.
  arXiv  Detail & Related papers  (2024-02-08T13:42:50Z)
• FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models [11.517609196300217]
  We introduce FuzzLLM, an automated fuzzing framework designed to proactively test and discover jailbreak vulnerabilities in Large Language Models (LLMs) We utilize templates to capture the structural integrity of a prompt and isolate key features of a jailbreak class as constraints. By integrating different base classes into powerful combo attacks and varying the elements of constraints and prohibited questions, FuzzLLM enables efficient testing with reduced manual effort.
  arXiv  Detail & Related papers  (2023-09-11T07:15:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.