Safeguard is a Double-edged Sword: Denial-of-service Attack on Large Language Models

• URL: http://arxiv.org/abs/2410.02916v2
• Date: Wed, 23 Oct 2024 17:26:06 GMT
• Title: Safeguard is a Double-edged Sword: Denial-of-service Attack on Large Language Models
• Authors: Qingzhao Zhang, Ziyang Xiong, Z. Morley Mao,
• Abstract summary: We present a new denial-of-service (DoS) attack on large language models (LLMs) By software or phishing attacks on user client software, attackers insert a short, seemingly innocuous adversarial prompt into to user prompt templates in configuration files. Our attack can automatically generate seemingly safe adversarial prompts, approximately only 30 characters long, that universally block over 97% of user requests on Llama Guard 3.
• Score: 7.013820690538764
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safety is a paramount concern of large language models (LLMs) in their open deployment. To this end, safeguard methods aim to enforce the ethical and responsible use of LLMs through safety alignment or guardrail mechanisms. However, we found that the malicious attackers could exploit false positives of safeguards, i.e., fooling the safeguard model to block safe content mistakenly, leading to a new denial-of-service (DoS) attack on LLMs. Specifically, by software or phishing attacks on user client software, attackers insert a short, seemingly innocuous adversarial prompt into to user prompt templates in configuration files; thus, this prompt appears in final user requests without visibility in the user interface and is not trivial to identify. By designing an optimization process that utilizes gradient and attention information, our attack can automatically generate seemingly safe adversarial prompts, approximately only 30 characters long, that universally block over 97\% of user requests on Llama Guard 3. The attack presents a new dimension of evaluating LLM safeguards focusing on false positives, fundamentally different from the classic jailbreak.

Related papers

• FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks [45.65210717380502]
  Large language models (LLMs) have been widely deployed as the backbone with additional tools and text information for real-world applications. prompt injection attacks are particularly threatening, where malicious instructions injected in the external text information can exploit LLMs to generate answers as the attackers desire. This paper introduces a novel test-time defense strategy, named AuThentication with Hash-based tags (FATH)
  arXiv  Detail & Related papers  (2024-10-28T20:02:47Z)
• SPIN: Self-Supervised Prompt INjection [16.253558670549697]
  adversarial and jailbreak attacks have been proposed to bypass the safety alignment and cause the model to produce harmful responses. We introduce Self-supervised Prompt INjection (SPIN) which can detect and reverse these various attacks on LLMs. Our system can reduce the attack success rate by up to 87.9%, while maintaining the performance on benign user requests.
  arXiv  Detail & Related papers  (2024-10-17T05:40:54Z)
• MoJE: Mixture of Jailbreak Experts, Naive Tabular Classifiers as Guard for Prompt Attacks [2.873719680183099]
  This paper advocates for the significance of jailbreak attack prevention on Large Language Models (LLMs) We introduce MoJE, a novel guardrail architecture designed to surpass current limitations in existing state-of-the-art guardrails. MoJE excels in detecting jailbreak attacks while maintaining minimal computational overhead during model inference.
  arXiv  Detail & Related papers  (2024-09-26T10:12:19Z)
• Purple-teaming LLMs with Adversarial Defender Training [57.535241000787416]
  We present Purple-teaming LLMs with Adversarial Defender training (PAD) PAD is a pipeline designed to safeguard LLMs by novelly incorporating the red-teaming (attack) and blue-teaming (safety training) techniques. PAD significantly outperforms existing baselines in both finding effective attacks and establishing a robust safe guardrail.
  arXiv  Detail & Related papers  (2024-07-01T23:25:30Z)
• Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks [59.46556573924901]
  This paper introduces Defensive Prompt Patch (DPP), a novel prompt-based defense mechanism for large language models (LLMs) Unlike previous approaches, DPP is designed to achieve a minimal Attack Success Rate (ASR) while preserving the high utility of LLMs. Empirical results conducted on LLAMA-2-7B-Chat and Mistral-7B-Instruct-v0.2 models demonstrate the robustness and adaptability of DPP.
  arXiv  Detail & Related papers  (2024-05-30T14:40:35Z)
• LeapFrog: The Rowhammer Instruction Skip Attack [4.091772241106195]
  We present a new type of Rowhammer gadget, called a LeapFrog gadget, which allows an adversary to subvert code execution. The Leapfrog gadget manifests when the victim code stores the Program Counter (PC) value in the user or kernel stack. This research also presents a systematic process to identify Leapfrog gadgets.
  arXiv  Detail & Related papers  (2024-04-11T16:10:16Z)
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting [54.931241667414184]
  We propose textbfAdaptive textbfShield Prompting, which prepends inputs with defense prompts to defend MLLMs against structure-based jailbreak attacks. Our methods can consistently improve MLLMs' robustness against structure-based jailbreak attacks.
  arXiv  Detail & Related papers  (2024-03-14T15:57:13Z)
• AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models [55.748851471119906]
  Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against these attacks is possible: adversarial attacks generate unlimited but unreadable gibberish prompts, detectable by perplexity-based filters. We introduce AutoDAN, an interpretable, gradient-based adversarial attack that merges the strengths of both attack types.
  arXiv  Detail & Related papers  (2023-10-23T17:46:07Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.