Understanding and Enhancing the Transferability of Jailbreaking Attacks

• URL: http://arxiv.org/abs/2502.03052v1
• Date: Wed, 05 Feb 2025 10:29:54 GMT
• Title: Understanding and Enhancing the Transferability of Jailbreaking Attacks
• Authors: Runqi Lin, Bo Han, Fengwang Li, Tongling Liu,
• Abstract summary: Jailbreaking attacks can effectively manipulate open-source large language models (LLMs) to produce harmful responses. This work investigates the transferability of jailbreaking attacks by analysing their impact on the model's intent perception. We propose the Perceived-importance Flatten (PiF) method, which uniformly disperses the model's focus across neutral-intent tokens in the original input.
• Score: 12.446931518819875
• License:
• Abstract: Jailbreaking attacks can effectively manipulate open-source large language models (LLMs) to produce harmful responses. However, these attacks exhibit limited transferability, failing to disrupt proprietary LLMs consistently. To reliably identify vulnerabilities in proprietary LLMs, this work investigates the transferability of jailbreaking attacks by analysing their impact on the model's intent perception. By incorporating adversarial sequences, these attacks can redirect the source LLM's focus away from malicious-intent tokens in the original input, thereby obstructing the model's intent recognition and eliciting harmful responses. Nevertheless, these adversarial sequences fail to mislead the target LLM's intent perception, allowing the target LLM to refocus on malicious-intent tokens and abstain from responding. Our analysis further reveals the inherent distributional dependency within the generated adversarial sequences, whose effectiveness stems from overfitting the source LLM's parameters, resulting in limited transferability to target LLMs. To this end, we propose the Perceived-importance Flatten (PiF) method, which uniformly disperses the model's focus across neutral-intent tokens in the original input, thus obscuring malicious-intent tokens without relying on overfitted adversarial sequences. Extensive experiments demonstrate that PiF provides an effective and efficient red-teaming evaluation for proprietary LLMs.

Related papers

• Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLM [53.79753074854936]
  Large language models (LLMs) are increasingly vulnerable to emerging jailbreak attacks. This vulnerability poses significant risks to the real-world applications. We propose a novel defensive paradigm called GuidelineLLM.
  arXiv  Detail & Related papers  (2024-12-10T12:42:33Z)
• Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation [4.241100280846233]
  AI agents, powered by large language models (LLMs), have transformed human-computer interactions by enabling seamless, natural, and context-aware communication. This paper investigates a critical vulnerability: adversarial attacks targeting the LLM core within AI agents.
  arXiv  Detail & Related papers  (2024-12-05T18:38:30Z)
• Robust LLM safeguarding via refusal feature adversarial training [15.76605079209956]
  Large language models (LLMs) are vulnerable to adversarial attacks that can elicit harmful responses. We propose Refusal Feature Adrial Training (ReFAT), a novel algorithm that efficiently performs adversarial training. Experiment results show that ReFAT significantly improves the robustness of three popular LLMs against a wide range of adversarial attacks.
  arXiv  Detail & Related papers  (2024-09-30T08:41:39Z)
• Jailbreaking as a Reward Misspecification Problem [80.52431374743998]
  We propose a novel perspective that attributes this vulnerability to reward misspecification during the alignment process. We introduce a metric ReGap to quantify the extent of reward misspecification and demonstrate its effectiveness. We present ReMiss, a system for automated red teaming that generates adversarial prompts in a reward-misspecified space.
  arXiv  Detail & Related papers  (2024-06-20T15:12:27Z)
• Chain-of-Scrutiny: Detecting Backdoor Attacks for Large Language Models [35.77228114378362]
  Large Language Models (LLMs) generate malicious outputs when inputs contain specific "triggers" set by attackers. Traditional defense strategies are impractical for API-accessible LLMs due to limited model access, high computational costs, and data requirements. We propose Chain-of-Scrutiny (CoS) which leverages LLMs' unique reasoning abilities to mitigate backdoor attacks.
  arXiv  Detail & Related papers  (2024-06-10T00:53:25Z)
• ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings [58.82536530615557]
  We propose an Adversarial Suffix Embedding Translation Framework (ASETF) to transform continuous adversarial suffix embeddings into coherent and understandable text. Our method significantly reduces the computation time of adversarial suffixes and achieves a much better attack success rate to existing techniques.
  arXiv  Detail & Related papers  (2024-02-25T06:46:27Z)
• Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective [43.94115802328438]
  Recent surge in jailbreaking attacks has revealed significant vulnerabilities in Large Language Models (LLMs) when exposed to malicious inputs. We suggest that the self-safeguarding capability of LLMs is linked to specific activity patterns within their representation space. Our findings demonstrate that these patterns can be detected with just a few pairs of contrastive queries.
  arXiv  Detail & Related papers  (2024-01-12T00:50:04Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities. Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content. We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Hijacking Large Language Models via Adversarial In-Context Learning [8.15194326639149]
  In-context learning (ICL) has emerged as a powerful paradigm leveraging LLMs for specific downstream tasks. Existing attacks are either easy to detect, rely on external models, or lack specificity towards ICL. This work introduces a novel transferable attack against ICL to address these issues.
  arXiv  Detail & Related papers  (2023-11-16T15:01:48Z)
• MART: Improving LLM Safety with Multi-round Automatic Red-Teaming [72.2127916030909]
  We propose a Multi-round Automatic Red-Teaming (MART) method, which incorporates both automatic adversarial prompt writing and safe response generation. On adversarial prompt benchmarks, the violation rate of an LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART. Notably, model helpfulness on non-adversarial prompts remains stable throughout iterations, indicating the target LLM maintains strong performance on instruction following.
  arXiv  Detail & Related papers  (2023-11-13T19:13:29Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.