Investigating Adversarial Trigger Transfer in Large Language Models

• URL: http://arxiv.org/abs/2404.16020v2
• Date: Tue, 08 Apr 2025 22:19:29 GMT
• Title: Investigating Adversarial Trigger Transfer in Large Language Models
• Authors: Nicholas Meade, Arkil Patel, Siva Reddy,
• Abstract summary: We show that adversarial triggers are not consistently transferable.<n>We show that APO models are extremely hard to jailbreak even when the trigger is optimized directly on the model.<n>Most triggers optimized on AFT models also generalize to new unsafe instructions from five diverse domains.
• Score: 28.97206621629125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has developed optimization procedures to find token sequences, called adversarial triggers, which can elicit unsafe responses from aligned language models. These triggers are believed to be highly transferable, i.e., a trigger optimized on one model can jailbreak other models. In this paper, we concretely show that such adversarial triggers are not consistently transferable. We extensively investigate trigger transfer amongst 13 open models and observe poor and inconsistent transfer. Our experiments further reveal a significant difference in robustness to adversarial triggers between models Aligned by Preference Optimization (APO) and models Aligned by Fine-Tuning (AFT). We find that APO models are extremely hard to jailbreak even when the trigger is optimized directly on the model. On the other hand, while AFT models may appear safe on the surface, exhibiting refusals to a range of unsafe instructions, we show that they are highly susceptible to adversarial triggers. Lastly, we observe that most triggers optimized on AFT models also generalize to new unsafe instructions from five diverse domains, further emphasizing their vulnerability. Overall, our work highlights the need for more comprehensive safety evaluations for aligned language models.

Related papers

• Towards Model Resistant to Transferable Adversarial Examples via Trigger Activation [95.3977252782181]
  Adversarial examples, characterized by imperceptible perturbations, pose significant threats to deep neural networks by misleading their predictions. We introduce a novel training paradigm aimed at enhancing robustness against transferable adversarial examples (TAEs) in a more efficient and effective way.
  arXiv  Detail & Related papers  (2025-04-20T09:07:10Z)
• SafeRoute: Adaptive Model Selection for Efficient and Accurate Safety Guardrails in Large Language Models [63.63254955809224]
  We propose a binary router that distinguishes hard examples from easy ones. Our method selectively applies the larger safety guard model to the data that the router considers hard, improving efficiency while maintaining accuracy. Experimental results on multiple benchmark datasets demonstrate that our adaptive model selection significantly enhances the trade-off between computational cost and safety performance.
  arXiv  Detail & Related papers  (2025-02-18T02:51:17Z)
• DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing [62.43110639295449]
  Large Language Models (LLMs) are widely applied in decision making, but their deployment is threatened by jailbreak attacks. Delman is a novel approach leveraging direct model editing for precise, dynamic protection against jailbreak attacks. Delman directly updates a minimal set of relevant parameters to neutralize harmful behaviors while preserving the model's utility.
  arXiv  Detail & Related papers  (2025-02-17T10:39:21Z)
• A Realistic Threat Model for Large Language Model Jailbreaks [87.64278063236847]
  In this work, we propose a unified threat model for the principled comparison of jailbreak attacks. Our threat model combines constraints in perplexity, measuring how far a jailbreak deviates from natural text. We adapt popular attacks to this new, realistic threat model, with which we, for the first time, benchmark these attacks on equal footing.
  arXiv  Detail & Related papers  (2024-10-21T17:27:01Z)
• TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors [36.07978634674072]
  Diffusion models are vulnerable to backdoor attacks that compromise their integrity. We propose TERD, a backdoor defense framework that builds unified modeling for current attacks. TERD secures a 100% True Positive Rate (TPR) and True Negative Rate (TNR) across datasets of varying resolutions.
  arXiv  Detail & Related papers  (2024-09-09T03:02:16Z)
• What Makes and Breaks Safety Fine-tuning? A Mechanistic Study [64.9691741899956]
  Safety fine-tuning helps align Large Language Models (LLMs) with human preferences for their safe deployment. We design a synthetic data generation framework that captures salient aspects of an unsafe input. Using this, we investigate three well-known safety fine-tuning methods.
  arXiv  Detail & Related papers  (2024-07-14T16:12:57Z)
• Watch Out for Your Guidance on Generation! Exploring Conditional Backdoor Attacks against Large Language Models [8.348993615202138]
  backdoor attacks on large language models (LLMs) typically set a fixed trigger in the input instance and specific responses for triggered queries. We present a new poisoning paradigm against LLMs triggered by specifying generation conditions. The poisoned model performs normally for output under normal/other generation conditions, while becoming harmful for output under target generation conditions.
  arXiv  Detail & Related papers  (2024-04-23T07:19:20Z)
• Query-Based Adversarial Prompt Generation [67.238873588125]
  We build adversarial examples that cause an aligned language model to emit harmful strings. We validate our attack on GPT-3.5 and OpenAI's safety classifier.
  arXiv  Detail & Related papers  (2024-02-19T18:01:36Z)
• FTA: Stealthy and Adaptive Backdoor Attack with Flexible Triggers on Federated Learning [11.636353298724574]
  We propose a new stealthy and robust backdoor attack against federated learning (FL) defenses. We build a generative trigger function that can learn to manipulate benign samples with an imperceptible flexible trigger pattern. Our trigger generator can keep learning and adapt across different rounds, allowing it to adjust to changes in the global model.
  arXiv  Detail & Related papers  (2023-08-31T20:25:54Z)
• Exploring the Universal Vulnerability of Prompt-based Learning Paradigm [21.113683206722207]
  We find that prompt-based learning bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. We explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text.
  arXiv  Detail & Related papers  (2022-04-11T16:34:10Z)
• MINIMAL: Mining Models for Data Free Universal Adversarial Triggers [57.14359126600029]
  We present a novel data-free approach, MINIMAL, to mine input-agnostic adversarial triggers from NLP models. We reduce the accuracy of Stanford Sentiment Treebank's positive class from 93.6% to 9.6%. For the Stanford Natural Language Inference (SNLI), our single-word trigger reduces the accuracy of the entailment class from 90.95% to less than 0.6%.
  arXiv  Detail & Related papers  (2021-09-25T17:24:48Z)
• "What's in the box?!": Deflecting Adversarial Attacks by Randomly Deploying Adversarially-Disjoint Models [71.91835408379602]
  adversarial examples have been long considered a real threat to machine learning models. We propose an alternative deployment-based defense paradigm that goes beyond the traditional white-box and black-box threat models.
  arXiv  Detail & Related papers  (2021-02-09T20:07:13Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.