Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks

• URL: http://arxiv.org/abs/2508.20038v3
• Date: Thu, 04 Sep 2025 09:23:46 GMT
• Title: Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attacks
• Authors: Sheng Liu, Qiang Sheng, Danding Wang, Yang Li, Guang Yang, Juan Cao,
• Abstract summary: New attacks expose large language models' inability to recognize unseen malicious instructions.<n>We propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions.<n>We show significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.
• Score: 29.465042445657947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite advances in improving large language model (LLM) to refuse to answer malicious instructions, widely used LLMs remain vulnerable to jailbreak attacks where attackers generate instructions with distributions differing from safety alignment corpora. New attacks expose LLMs' inability to recognize unseen malicious instructions, highlighting a critical distributional mismatch between training data and real-world attacks that forces developers into reactive patching cycles. To tackle this challenge, we propose IMAGINE, a synthesis framework that leverages embedding space distribution analysis to generate jailbreak-like instructions. This approach effectively fills the distributional gap between authentic jailbreak patterns and safety alignment corpora. IMAGINE follows an iterative optimization process that dynamically evolves text generation distributions across iterations, thereby augmenting the coverage of safety alignment data distributions through synthesized data examples. Based on the safety-aligned corpus enhanced through IMAGINE, our framework demonstrates significant decreases in attack success rate on Qwen2.5, Llama3.1, and Llama3.2 without compromising their utility.

Related papers

• Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning [48.100552417137656]
  PASS employs reinforcement learning to transform initial jailbreak prompts into formalized descriptions.<n>We conduct extensive experiments on common open-source models, demonstrating the effectiveness of our attack.
  arXiv  Detail & Related papers  (2025-09-28T01:38:00Z)
• The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMs [39.85609149662187]
  We present DIJA, the first systematic study and jailbreak attack framework that exploits unique safety weaknesses of dLLMs.<n>Our proposed DIJA constructs adversarial interleaved mask-text prompts that exploit the text generation mechanisms of dLLMs.<n>Our findings underscore the urgent need for rethinking safety alignment in this emerging class of language models.
  arXiv  Detail & Related papers  (2025-07-15T08:44:46Z)
• Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models [6.049325292667881]
  We present a systematic black-box security analysis of prefill-level jailbreak attacks.<n>Our experiments show that prefill-level attacks achieve high success rates, with adaptive methods exceeding 99% on several models.<n>We find that a detection method focusing on the manipulative relationship between the prompt and the prefill is more effective.
  arXiv  Detail & Related papers  (2025-04-28T07:38:43Z)
• Improving LLM Safety Alignment with Dual-Objective Optimization [65.41451412400609]
  Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
  arXiv  Detail & Related papers  (2025-03-05T18:01:05Z)
• xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM Jailbreaking [32.89084809038529]
  Black-box jailbreak is an attack where crafted prompts bypass safety mechanisms in large language models.<n>We propose a novel black-box jailbreak method leveraging reinforcement learning (RL)<n>We introduce a comprehensive jailbreak evaluation framework incorporating keywords, intent matching, and answer validation to provide a more rigorous and holistic assessment of jailbreak success.
  arXiv  Detail & Related papers  (2025-01-28T06:07:58Z)
• h4rm3l: A language for Composable Jailbreak Attack Synthesis [48.5611060845958]
  h4rm3l is a novel approach that addresses the gap with a human-readable domain-specific language.<n>We show that h4rm3l's synthesized attacks are diverse and more successful than existing jailbreak attacks in literature.
  arXiv  Detail & Related papers  (2024-08-09T01:45:39Z)
• AutoJailbreak: Exploring Jailbreak Attacks and Defenses through a Dependency Lens [83.08119913279488]
  We present a systematic analysis of the dependency relationships in jailbreak attack and defense techniques. We propose three comprehensive, automated, and logical frameworks. We show that the proposed ensemble jailbreak attack and defense framework significantly outperforms existing research.
  arXiv  Detail & Related papers  (2024-06-06T07:24:41Z)
• Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks [59.46556573924901]
  This paper introduces Defensive Prompt Patch (DPP), a novel prompt-based defense mechanism for large language models (LLMs)<n>Unlike previous approaches, DPP is designed to achieve a minimal Attack Success Rate (ASR) while preserving the high utility of LLMs.<n> Empirical results conducted on LLAMA-2-7B-Chat and Mistral-7B-Instruct-v0.2 models demonstrate the robustness and adaptability of DPP.
  arXiv  Detail & Related papers  (2024-05-30T14:40:35Z)
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting [54.931241667414184]
  We propose textbfAdaptive textbfShield Prompting, which prepends inputs with defense prompts to defend MLLMs against structure-based jailbreak attacks. Our methods can consistently improve MLLMs' robustness against structure-based jailbreak attacks.
  arXiv  Detail & Related papers  (2024-03-14T15:57:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.