How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation

• URL: http://arxiv.org/abs/2502.14486v1
• Date: Thu, 20 Feb 2025 12:07:40 GMT
• Title: How Jailbreak Defenses Work and Ensemble? A Mechanistic Investigation
• Authors: Zhuohang Long, Siyuan Wang, Shujun Liu, Yuhang Lai, Xuanjing Huang, Zhongyu Wei,
• Abstract summary: Jailbreak attacks, where harmful prompts bypass generative models' built-in safety, raise serious concerns about model vulnerability.<n>This paper systematically examines jailbreak defenses by reframing the standard generation task as a binary classification problem.<n>We identify two key defense mechanisms: safety shift, which increases refusal rates across all queries, and harmfulness discrimination, which improves the model's ability to distinguish between harmful and benign inputs.
• Score: 39.44000290664494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Jailbreak attacks, where harmful prompts bypass generative models' built-in safety, raise serious concerns about model vulnerability. While many defense methods have been proposed, the trade-offs between safety and helpfulness, and their application to Large Vision-Language Models (LVLMs), are not well understood. This paper systematically examines jailbreak defenses by reframing the standard generation task as a binary classification problem to assess model refusal tendencies for both harmful and benign queries. We identify two key defense mechanisms: safety shift, which increases refusal rates across all queries, and harmfulness discrimination, which improves the model's ability to distinguish between harmful and benign inputs. Using these mechanisms, we develop two ensemble defense strategies-inter-mechanism ensembles and intra-mechanism ensembles-to balance safety and helpfulness. Experiments on the MM-SafetyBench and MOSSBench datasets with LLaVA-1.5 models show that these strategies effectively improve model safety or optimize the trade-off between safety and helpfulness.

Related papers

• Secure Tug-of-War (SecTOW): Iterative Defense-Attack Training with Reinforcement Learning for Multimodal Model Security [63.41350337821108]
  We propose Secure Tug-of-War (SecTOW) to enhance the security of multimodal large language models (MLLMs)<n>SecTOW consists of two modules: a defender and an auxiliary attacker, both trained iteratively using reinforcement learning (GRPO)<n>We show that SecTOW significantly improves security while preserving general performance.
  arXiv  Detail & Related papers  (2025-07-29T17:39:48Z)
• The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models [5.984437476321095]
  We open-source a toolkit to estimate the safety gap for state-of-the-art open-weight models.<n>We evaluate biochemical and cyber capabilities, refusal rates, and generation quality of models from two families.<n>Our experiments reveal that the safety gap widens as model scale increases and effective dangerous capabilities grow substantially when safeguards are removed.
  arXiv  Detail & Related papers  (2025-07-08T23:58:01Z)
• Chain-of-Lure: A Synthetic Narrative-Driven Approach to Compromise Large Language Models [15.134149399922192]
  We propose a novel jailbreaking method inspired by the Chain-of-Thought mechanism.<n>The attacker model uses mission transfer to conceal harmful user intent in dialogue and generates chained narrative lures to stimulate the reasoning capabilities of victim models.<n>Our experiments reveal that models with weaker safety mechanisms exhibit stronger attack capabilities, demonstrating that models can not only be exploited, but also help harm others.
  arXiv  Detail & Related papers  (2025-05-23T06:19:05Z)
• SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning [76.56522719330911]
  Large Reasoning Models (LRMs) introduce a new generation paradigm of explicitly reasoning before answering.<n>LRMs pose great safety risks against harmful queries and adversarial attacks.<n>We propose SafeKey to better activate the safety aha moment in the key sentence.
  arXiv  Detail & Related papers  (2025-05-22T03:46:03Z)
• T2V-OptJail: Discrete Prompt Optimization for Text-to-Video Jailbreak Attacks [67.91652526657599]
  We formalize the T2V jailbreak attack as a discrete optimization problem and propose a joint objective-based optimization framework, called T2V-OptJail.<n>We conduct large-scale experiments on several T2V models, covering both open-source models and real commercial closed-source models.<n>The proposed method improves 11.4% and 10.0% over the existing state-of-the-art method in terms of attack success rate.
  arXiv  Detail & Related papers  (2025-05-10T16:04:52Z)
• T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Models [88.63040835652902]
  Text to video models are vulnerable to jailbreak attacks, where specially crafted prompts bypass safety mechanisms and lead to the generation of harmful or unsafe content. We propose T2VShield, a comprehensive and model agnostic defense framework designed to protect text to video models from jailbreak threats. Our method systematically analyzes the input, model, and output stages to identify the limitations of existing defenses.
  arXiv  Detail & Related papers  (2025-04-22T01:18:42Z)
• Evolving Security in LLMs: A Study of Jailbreak Attacks and Defenses [0.5261718469769449]
  Large Language Models (LLMs) are increasingly popular, powering a wide range of applications. Their widespread use has sparked concerns, especially through jailbreak attacks that bypass safety measures to produce harmful content. We present a comprehensive security analysis of large language models (LLMs), addressing critical research questions on the evolution and determinants of model safety.
  arXiv  Detail & Related papers  (2025-04-02T19:33:07Z)
• Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreaking [26.812138599896997]
  We propose Reasoning-to-Defend (R2D), a novel training paradigm that integrates a safety-aware reasoning mechanism into Large Language Models' generation.<n>R2D forms safety pivot tokens as indicators of the safety status of responses.<n>We show that R2D effectively mitigates various attacks and improves overall safety, while maintaining the original performances.
  arXiv  Detail & Related papers  (2025-02-18T15:48:46Z)
• Robustness of Large Language Models Against Adversarial Attacks [5.312946761836463]
  We present a comprehensive study on the robustness of GPT LLM family. We employ two distinct evaluation methods to assess their resilience. Our experiments reveal significant variations in the robustness of these models, demonstrating their varying degrees of vulnerability to both character-level and semantic-level adversarial attacks.
  arXiv  Detail & Related papers  (2024-12-22T13:21:15Z)
• The Great Contradiction Showdown: How Jailbreak and Stealth Wrestle in Vision-Language Models? [23.347349690954452]
  Vision-Language Models (VLMs) have achieved remarkable performance on a variety of tasks, yet they remain vulnerable to jailbreak attacks. We provide an information-theoretic framework for understanding the fundamental trade-off between the effectiveness of these attacks and their stealthiness. We propose an efficient algorithm for detecting non-stealthy jailbreak attacks, offering significant improvements in model robustness.
  arXiv  Detail & Related papers  (2024-10-02T11:40:49Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance [48.36220909956064]
  SafeAligner is a methodology implemented at the decoding stage to fortify defenses against jailbreak attacks.<n>We develop two specialized models: the Sentinel Model, which is trained to foster safety, and the Intruder Model, designed to generate riskier responses.<n>We show that SafeAligner can increase the likelihood of beneficial tokens, while reducing the occurrence of harmful ones.
  arXiv  Detail & Related papers  (2024-06-26T07:15:44Z)
• Watch the Watcher! Backdoor Attacks on Security-Enhancing Diffusion Models [65.30406788716104]
  This work investigates the vulnerabilities of security-enhancing diffusion models. We demonstrate that these models are highly susceptible to DIFF2, a simple yet effective backdoor attack. Case studies show that DIFF2 can significantly reduce both post-purification and certified accuracy across benchmark datasets and models.
  arXiv  Detail & Related papers  (2024-06-14T02:39:43Z)
• Jailbroken: How Does LLM Safety Training Fail? [92.8748773632051]
  "jailbreak" attacks on early releases of ChatGPT elicit undesired behavior. We investigate why such attacks succeed and how they can be created. New attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests.
  arXiv  Detail & Related papers  (2023-07-05T17:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.