AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs

• URL: http://arxiv.org/abs/2404.16873v2
• Date: Mon, 02 Jun 2025 18:59:01 GMT
• Title: AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
• Authors: Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, Yuandong Tian,
• Abstract summary: Large Language Models (LLMs) are vulnerable to jailbreaking attacks that lead to generation of inappropriate or harmful content.<n>We present a novel method that uses another LLM, called AdvPrompter, to generate human-readable adversarial prompts in seconds.
• Score: 51.217126257318924
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are vulnerable to jailbreaking attacks that lead to generation of inappropriate or harmful content. Manual red-teaming requires a time-consuming search for adversarial prompts, whereas automatic adversarial prompt generation often leads to semantically meaningless attacks that do not scale well. In this paper, we present a novel method that uses another LLM, called AdvPrompter, to generate human-readable adversarial prompts in seconds. AdvPrompter, which is trained using an alternating optimization algorithm, generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response. Experimental results on popular open source TargetLLMs show highly competitive results on the AdvBench and HarmBench datasets, that also transfer to closed-source black-box LLMs. We also show that training on adversarial suffixes generated by AdvPrompter is a promising strategy for improving the robustness of LLMs to jailbreaking attacks.

Related papers

• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction [68.6543680065379]
  Large language models (LLMs) are vulnerable to prompt injection attacks. We propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-04-29T07:13:53Z)
• Computing Optimization-Based Prompt Injections Against Closed-Weights Models By Misusing a Fine-Tuning API [3.908034401768844]
  We describe how an attacker can leverage the loss-like information returned from the remote fine-tuning interface to guide the search for adversarial prompts. We demonstrate attack success rates between 65% and 82% on Google's Gemini family of LLMs.
  arXiv  Detail & Related papers  (2025-01-16T19:01:25Z)
• GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs [3.096869664709865]
  We introduce Generative Adversarial Suffix Prompter (GASP) to improve adversarial suffix creation in a fully black-box setting. Our experiments show that GASP can generate natural jailbreak prompts, significantly improving attack success rates, reducing training times, and accelerating inference speed.
  arXiv  Detail & Related papers  (2024-11-21T14:00:01Z)
• Aligning LLMs to Be Robust Against Prompt Injection [55.07562650579068]
  We show that alignment can be a powerful tool to make LLMs more robust against prompt injection attacks. Our method -- SecAlign -- first builds an alignment dataset by simulating prompt injection attacks. Our experiments show that SecAlign robustifies the LLM substantially with a negligible hurt on model utility.
  arXiv  Detail & Related papers  (2024-10-07T19:34:35Z)
• Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context [49.13497493053742]
  This research explores converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs.
  arXiv  Detail & Related papers  (2024-07-19T19:47:26Z)
• Defending Against Indirect Prompt Injection Attacks With Spotlighting [11.127479817618692]
  In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input.
  arXiv  Detail & Related papers  (2024-03-20T15:26:23Z)
• DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers [74.7446827091938]
  We introduce an automatic prompt textbfDecomposition and textbfReconstruction framework for jailbreak textbfAttack (DrAttack) DrAttack includes three key components: (a) Decomposition' of the original prompt into sub-prompts, (b) Reconstruction' of these sub-prompts implicitly by in-context learning with semantically similar but harmless reassembling demo, and (c) a Synonym Search' of sub-prompts, aiming to find sub-prompts' synonyms that maintain the original intent while
  arXiv  Detail & Related papers  (2024-02-25T17:43:29Z)
• ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings [58.82536530615557]
  We propose an Adversarial Suffix Embedding Translation Framework (ASETF) to transform continuous adversarial suffix embeddings into coherent and understandable text. Our method significantly reduces the computation time of adversarial suffixes and achieves a much better attack success rate to existing techniques.
  arXiv  Detail & Related papers  (2024-02-25T06:46:27Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)
• Detecting Language Model Attacks with Perplexity [0.0]
  A novel hack involving Large Language Models (LLMs) has emerged, exploiting adversarial suffixes to deceive models into generating perilous responses. A Light-GBM trained on perplexity and token length resolved the false positives and correctly detected most adversarial attacks in the test set.
  arXiv  Detail & Related papers  (2023-08-27T15:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.