Jailbroken: How Does LLM Safety Training Fail?
- URL: http://arxiv.org/abs/2307.02483v1
- Date: Wed, 5 Jul 2023 17:58:10 GMT
- Title: Jailbroken: How Does LLM Safety Training Fail?
- Authors: Alexander Wei and Nika Haghtalab and Jacob Steinhardt
- Abstract summary: "jailbreak" attacks on early releases of ChatGPT elicit undesired behavior.
We investigate why such attacks succeed and how they can be created.
New attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests.
- Score: 92.8748773632051
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large language models trained for safety and harmlessness remain susceptible
to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on
early releases of ChatGPT that elicit undesired behavior. Going beyond
recognition of the issue, we investigate why such attacks succeed and how they
can be created. We hypothesize two failure modes of safety training: competing
objectives and mismatched generalization. Competing objectives arise when a
model's capabilities and safety goals conflict, while mismatched generalization
occurs when safety training fails to generalize to a domain for which
capabilities exist. We use these failure modes to guide jailbreak design and
then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's
Claude v1.3, against both existing and newly designed attacks. We find that
vulnerabilities persist despite the extensive red-teaming and safety-training
efforts behind these models. Notably, new attacks utilizing our failure modes
succeed on every prompt in a collection of unsafe requests from the models'
red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our
analysis emphasizes the need for safety-capability parity -- that safety
mechanisms should be as sophisticated as the underlying model -- and argues
against the idea that scaling alone can resolve these safety failure modes.
Related papers
- Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training [67.30423823744506]
This study addresses a critical gap in safety tuning practices for Large Language Models (LLMs)
We introduce a novel approach, Decoupled Refusal Training (DeRTa), designed to empower LLMs to refuse compliance to harmful prompts at any response position.
DeRTa incorporates two novel components: (1) Maximum Likelihood Estimation with Harmful Response Prefix, which trains models to recognize and avoid unsafe content by appending a segment of harmful response to the beginning of a safe response, and (2) Reinforced Transition Optimization (RTO), which equips models with the ability to transition from potential harm to safety refusal consistently throughout the harmful
arXiv Detail & Related papers (2024-07-12T09:36:33Z) - Jailbreak Attacks and Defenses Against Large Language Models: A Survey [22.392989536664288]
Large Language Models (LLMs) have performed exceptionally in various text-generative tasks.
"jailbreaking" induces the model to generate malicious responses against the usage policy and society.
We propose a comprehensive and detailed taxonomy of jailbreak attack and defense methods.
arXiv Detail & Related papers (2024-07-05T06:57:30Z) - A False Sense of Safety: Unsafe Information Leakage in 'Safe' AI Responses [42.136793654338106]
Large Language Models (LLMs) are vulnerable to leakages at$x2013$x2013$methods.
We introduce an inferential threat model called inferential adversaries who exploit impermissible information to achieve malicious goals.
Our work provides the first theoretically grounded understanding of the requirements for releasing safe jailbreaks and the utility costs involved.
arXiv Detail & Related papers (2024-07-02T16:19:25Z) - WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models [66.34505141027624]
We introduce WildTeaming, an automatic LLM safety red-teaming framework that mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of novel jailbreak tactics.
WildTeaming reveals previously unidentified vulnerabilities of frontier LLMs, resulting in up to 4.6x more diverse and successful adversarial attacks.
arXiv Detail & Related papers (2024-06-26T17:31:22Z) - SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance [48.80398992974831]
SafeAligner is a methodology implemented at the decoding stage to fortify defenses against jailbreak attacks.
We develop two specialized models: the Sentinel Model, which is trained to foster safety, and the Intruder Model, designed to generate riskier responses.
We show that SafeAligner can increase the likelihood of beneficial tokens, while reducing the occurrence of harmful ones.
arXiv Detail & Related papers (2024-06-26T07:15:44Z) - AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens [83.08119913279488]
We present a systematic analysis of the dependency relationships in jailbreak attack and defense techniques.
We propose three comprehensive, automated, and logical frameworks.
We show that the proposed ensemble jailbreak attack and defense framework significantly outperforms existing research.
arXiv Detail & Related papers (2024-06-06T07:24:41Z) - Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks [59.46556573924901]
This paper introduces Defensive Prompt Patch (DPP), a novel prompt-based defense mechanism for large language models (LLMs)
Unlike previous approaches, DPP is designed to achieve a minimal Attack Success Rate (ASR) while preserving the high utility of LLMs.
Empirical results conducted on LLAMA-2-7B-Chat and Mistral-7B-Instruct-v0.2 models demonstrate the robustness and adaptability of DPP.
arXiv Detail & Related papers (2024-05-30T14:40:35Z) - Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Review [15.179940846141873]
Applicating third-party data and models has become a new paradigm for language modeling in NLP.
backdoor attacks can induce the model to exhibit expected behaviors through specific triggers.
There is still no systematic and comprehensive review to reflect the security challenges, attacker's capabilities, and purposes.
arXiv Detail & Related papers (2023-09-12T08:48:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.