ASTPrompter: Weakly Supervised Automated Language Model Red-Teaming to Identify Likely Toxic Prompts

• URL: http://arxiv.org/abs/2407.09447v2
• Date: Fri, 18 Oct 2024 21:14:46 GMT
• Title: ASTPrompter: Weakly Supervised Automated Language Model Red-Teaming to Identify Likely Toxic Prompts
• Authors: Amelia F. Hardy, Houjun Liu, Bernard Lange, Mykel J. Kochenderfer,
• Abstract summary: We propose a reinforcement learning formulation of the red-teaming task that allows us to discover prompts that trigger toxic outputs from a frozen defender. We show that our policy is capable of generating likely (low-perplexity) prompts that also trigger toxicity from GPT-2, GPT-2 XL, and TinyLlama defenders.
• Score: 33.774939728834156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Typical schemes for the automated red-teaming of large language models (LLMs) focus on discovering prompts that trigger a frozen language model (the defender) to generate toxic text. This often results in the prompting model (the adversary) producing text that is unintelligible and unlikely to arise. Here, we propose a reinforcement learning formulation of the LLM red-teaming task that allows us to discover prompts that both (1) trigger toxic outputs from a frozen defender and (2) have low perplexity as scored by that defender. We argue these cases are the most pertinent in a red-teaming setting because they are likely to arise during normal use of the defender model. We solve this formulation through a novel online and weakly supervised variant of Identity Preference Optimization (IPO) on GPT-2, GPT-2 XL, and TinyLlama defenders. We demonstrate that our policy is capable of generating likely (low-perplexity) prompts that also trigger toxicity from all of these architectures. Furthermore, we show that this policy outperforms baselines by producing attacks that are occur with higher probability and are more effective. Finally, we discuss our findings and the observed trade-offs between likelihood vs toxicity. Source code for this project is available for this project at: https://github.com/sisl/ASTPrompter/.

Related papers

• Generalized Adversarial Code-Suggestions: Exploiting Contexts of LLM-based Code-Completion [4.940253381814369]
  adversarial code-suggestions can be introduced via data poisoning and, thus, unknowingly by the model creators. In this paper, we provide a generalized formulation of such attacks, spawning and extending related work in this domain. The latter gives rise to novel and more flexible targeted attack-strategies, allowing the adversary to choose the most suitable trigger pattern for a specific user-group arbitrarily.
  arXiv  Detail & Related papers  (2024-10-14T14:06:05Z)
• The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs [8.449922248196705]
  We present a subtle yet effective poisoning attack via user-supplied prompts to penetrate alignment training protections. Our attack, even without explicit knowledge about the target LLMs in the black-box setting, subtly alters the reward feedback mechanism. By injecting 1% of these specially crafted prompts into the data, through malicious users, we demonstrate a toxicity score up to two times higher when a specific trigger word is used.
  arXiv  Detail & Related papers  (2024-09-01T17:40:04Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• Goal-guided Generative Prompt Injection Attack on Large Language Models [6.175969971471705]
  Large language models (LLMs) provide a strong foundation for large-scale user-oriented natural language tasks. A large number of users can easily inject adversarial text or instructions through the user interface. It is unclear how these strategies relate to the success rate of attacks and thus effectively improve model security.
  arXiv  Detail & Related papers  (2024-04-06T06:17:10Z)
• AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models [55.748851471119906]
  Safety alignment of Large Language Models (LLMs) can be compromised with manual jailbreak attacks and (automatic) adversarial attacks. Recent studies suggest that defending against these attacks is possible: adversarial attacks generate unlimited but unreadable gibberish prompts, detectable by perplexity-based filters. We introduce AutoDAN, an interpretable, gradient-based adversarial attack that merges the strengths of both attack types.
  arXiv  Detail & Related papers  (2023-10-23T17:46:07Z)
• Universal and Transferable Adversarial Attacks on Aligned Language Models [118.41733208825278]
  We propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable.
  arXiv  Detail & Related papers  (2023-07-27T17:49:12Z)
• Effective Prompt Extraction from Language Models [70.00099540536382]
  We present a framework for measuring the effectiveness of prompt extraction attacks. In experiments with 3 different sources of prompts and 11 underlying large language models, we find that simple text-based attacks can in fact reveal prompts with high probability. Our framework determines with high precision whether an extracted prompt is the actual secret prompt, rather than a model hallucination.
  arXiv  Detail & Related papers  (2023-07-13T16:15:08Z)
• Red Teaming Language Model Detectors with Language Models [114.36392560711022]
  Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
  arXiv  Detail & Related papers  (2023-05-31T10:08:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.