Augmented Adversarial Trigger Learning

• URL: http://arxiv.org/abs/2503.12339v1
• Date: Sun, 16 Mar 2025 03:20:52 GMT
• Title: Augmented Adversarial Trigger Learning
• Authors: Zhe Wang, Yanjun Qi,
• Abstract summary: We propose ATLA: Adversarial Trigger Learning with Augmented objectives.<n>We show that ATLA consistently outperforms current state-of-the-art techniques.
• Score: 14.365410701358579
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gradient optimization-based adversarial attack methods automate the learning of adversarial triggers to generate jailbreak prompts or leak system prompts. In this work, we take a closer look at the optimization objective of adversarial trigger learning and propose ATLA: Adversarial Trigger Learning with Augmented objectives. ATLA improves the negative log-likelihood loss used by previous studies into a weighted loss formulation that encourages the learned adversarial triggers to optimize more towards response format tokens. This enables ATLA to learn an adversarial trigger from just one query-response pair and the learned trigger generalizes well to other similar queries. We further design a variation to augment trigger optimization with an auxiliary loss that suppresses evasive responses. We showcase how to use ATLA to learn adversarial suffixes jailbreaking LLMs and to extract hidden system prompts. Empirically we demonstrate that ATLA consistently outperforms current state-of-the-art techniques, achieving nearly 100% success in attacking while requiring 80% fewer queries. ATLA learned jailbreak suffixes demonstrate high generalization to unseen queries and transfer well to new LLMs.

Related papers

• Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction [68.6543680065379]
  Large language models (LLMs) are vulnerable to prompt injection attacks. We propose a novel defense method that leverages, rather than suppresses, the instruction-following abilities of LLMs.
  arXiv  Detail & Related papers  (2025-04-29T07:13:53Z)
• Iterative Prompting with Persuasion Skills in Jailbreaking Large Language Models [2.1511703382556657]
  This study exploits large language models (LLMs) with an iterative prompting technique. We analyze the response patterns of LLMs, including GPT-3.5, GPT-4, LLaMa2, Vicuna, and ChatGLM. Persuading strategies enhance prompt effectiveness while maintaining consistency with malicious intent.
  arXiv  Detail & Related papers  (2025-03-26T08:40:46Z)
• Computing Optimization-Based Prompt Injections Against Closed-Weights Models By Misusing a Fine-Tuning API [3.908034401768844]
  We describe how an attacker can leverage the loss-like information returned from the remote fine-tuning interface to guide the search for adversarial prompts.<n>We demonstrate attack success rates between 65% and 82% on Google's Gemini family of LLMs.
  arXiv  Detail & Related papers  (2025-01-16T19:01:25Z)
• Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models [61.916827858666906]
  Large Language Models (LLMs) are increasingly being integrated into services such as ChatGPT to provide responses to user queries.<n>This paper proposes a method called Token Highlighter to inspect and mitigate the potential jailbreak threats in the user query.
  arXiv  Detail & Related papers  (2024-12-24T05:10:02Z)
• Enhancing Adversarial Attacks through Chain of Thought [0.0]
  gradient-based adversarial attacks are particularly effective against aligned large language models (LLMs) This paper proposes enhancing the universality of adversarial attacks by integrating CoT prompts with the greedy coordinate gradient (GCG) technique.
  arXiv  Detail & Related papers  (2024-10-29T06:54:00Z)
• AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs [51.217126257318924]
  We present a novel method that uses another Large Language Models, called the AdvPrompter, to generate human-readable adversarial prompts in seconds. We train the AdvPrompter using a novel algorithm that does not require access to the gradients of the TargetLLM. The trained AdvPrompter generates suffixes that veil the input instruction without changing its meaning, such that the TargetLLM is lured to give a harmful response.
  arXiv  Detail & Related papers  (2024-04-21T22:18:13Z)
• ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings [58.82536530615557]
  We propose an Adversarial Suffix Embedding Translation Framework (ASETF) to transform continuous adversarial suffix embeddings into coherent and understandable text. Our method significantly reduces the computation time of adversarial suffixes and achieves a much better attack success rate to existing techniques.
  arXiv  Detail & Related papers  (2024-02-25T06:46:27Z)
• Defending Jailbreak Prompts via In-Context Adversarial Game [34.83853184278604]
  We introduce the In-Context Adversarial Game (ICAG) for defending against jailbreaks without the need for fine-tuning.<n>Unlike traditional methods that rely on static datasets, ICAG employs an iterative process to enhance both the defense and attack agents.<n>Our empirical studies affirm ICAG's efficacy, where LLMs safeguarded by ICAG exhibit significantly reduced jailbreak success rates.
  arXiv  Detail & Related papers  (2024-02-20T17:04:06Z)
• Attacking Large Language Models with Projected Gradient Descent [49.19426387912186]
  Projected Gradient Descent (PGD) for adversarial prompts is up to one order of magnitude faster than state-of-the-art discrete optimization.<n>Our PGD for LLMs is up to one order of magnitude faster than state-of-the-art discrete optimization to achieve the same devastating attack results.
  arXiv  Detail & Related papers  (2024-02-14T13:13:26Z)
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks [99.23352758320945]
  We propose SmoothLLM, the first algorithm designed to mitigate jailbreaking attacks on large language models (LLMs) Based on our finding that adversarially-generated prompts are brittle to character-level changes, our defense first randomly perturbs multiple copies of a given input prompt, and then aggregates the corresponding predictions to detect adversarial inputs.
  arXiv  Detail & Related papers  (2023-10-05T17:01:53Z)
• ALMOST: Adversarial Learning to Mitigate Oracle-less ML Attacks via Synthesis Tuning [18.758747687330384]
  Oracle-less machine learning (ML) attacks have broken various logic locking schemes. We propose ALMOST, a framework for adversarial learning to mitigate oracle-less ML attacks via synthesis tuning.
  arXiv  Detail & Related papers  (2023-03-06T18:55:58Z)
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection [64.67495502772866]
  Large Language Models (LLMs) are increasingly being integrated into various applications. We show how attackers can override original instructions and employed controls using Prompt Injection attacks. We derive a comprehensive taxonomy from a computer security perspective to systematically investigate impacts and vulnerabilities.
  arXiv  Detail & Related papers  (2023-02-23T17:14:38Z)
• Adversarial Training with Complementary Labels: On the Benefit of Gradually Informative Attacks [119.38992029332883]
  Adversarial training with imperfect supervision is significant but receives limited attention. We propose a new learning strategy using gradually informative attacks. Experiments are conducted to demonstrate the effectiveness of our method on a range of benchmarked datasets.
  arXiv  Detail & Related papers  (2022-11-01T04:26:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.