Is poisoning a real threat to LLM alignment? Maybe more so than you think

• URL: http://arxiv.org/abs/2406.12091v4
• Date: Mon, 09 Jun 2025 14:00:06 GMT
• Title: Is poisoning a real threat to LLM alignment? Maybe more so than you think
• Abstract summary: We investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning.<n>We find that unlike PPO-based methods, which, when it comes to backdoor attacks, require at least 4% of the data to be poisoned to elicit harmful behavior, we exploit the true vulnerabilities of DPO more simply so we can poison the model with only as much as 0.5% of the data.
• Score: 33.583240150025574
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in Reinforcement Learning with Human Feedback (RLHF) have significantly impacted the alignment of Large Language Models (LLMs). The sensitivity of reinforcement learning algorithms such as Proximal Policy Optimization (PPO) has led to new line work on Direct Policy Optimization (DPO), which treats RLHF in a supervised learning framework. The increased practical use of these RLHF methods warrants an analysis of their vulnerabilities. In this work, we investigate the vulnerabilities of DPO to poisoning attacks under different scenarios and compare the effectiveness of preference poisoning, a first of its kind. We comprehensively analyze DPO's vulnerabilities under different types of attacks, i.e., backdoor and non-backdoor attacks, and different poisoning methods across a wide array of language models, i.e., LLama 7B, Mistral 7B, and Gemma 7B. We find that unlike PPO-based methods, which, when it comes to backdoor attacks, require at least 4\% of the data to be poisoned to elicit harmful behavior, we exploit the true vulnerabilities of DPO more simply so we can poison the model with only as much as 0.5\% of the data. We further investigate the potential reasons behind the vulnerability and how well this vulnerability translates into backdoor vs non-backdoor attacks.

Related papers

• P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs [49.908234151374785]
  During fine-tuning, large language models (LLMs) are increasingly vulnerable to data-poisoning backdoor attacks.<n>We propose Poison-to-Poison (P2P), a general and effective backdoor defense algorithm.<n>We show that P2P can neutralize malicious backdoors while preserving task performance.
  arXiv  Detail & Related papers  (2025-10-06T05:45:23Z)
• Wolf Hidden in Sheep's Conversations: Toward Harmless Data-Based Backdoor Attacks for Jailbreaking Large Language Models [69.11679786018206]
  Supervised fine-tuning (SFT) aligns large language models with human intent by training them on labeled task-specific data.<n>Recent studies have shown that malicious attackers can inject backdoors into these models by embedding triggers into the harmful question-answer pairs.<n>We propose a novel clean-data backdoor attack for jailbreaking LLMs.
  arXiv  Detail & Related papers  (2025-05-23T08:13:59Z)
• ELBA-Bench: An Efficient Learning Backdoor Attacks Benchmark for Large Language Models [55.93380086403591]
  Generative large language models are vulnerable to backdoor attacks. $textitELBA-Bench$ allows attackers to inject backdoor through parameter efficient fine-tuning. $textitELBA-Bench$ provides over 1300 experiments.
  arXiv  Detail & Related papers  (2025-02-22T12:55:28Z)
• PBP: Post-training Backdoor Purification for Malware Classifiers [5.112004957241861]
  In recent years, the rise of machine learning (ML) in cybersecurity has brought new challenges, including the increasing threat of backdoor poisoning attacks. Here, we introduce PBP, a post-training defense for malware classifiers that mitigates various types of backdoor embeddings without assuming any specific backdoor embedding mechanism. Our method demonstrates substantial advantages over several state-of-the-art methods, as evidenced by experiments on two datasets, two types of backdoor methods, and various attack configurations.
  arXiv  Detail & Related papers  (2024-12-04T16:30:03Z)
• Efficient Backdoor Defense in Multimodal Contrastive Learning: A Token-Level Unlearning Method for Mitigating Threats [52.94388672185062]
  We propose an efficient defense mechanism against backdoor threats using a concept known as machine unlearning. This entails strategically creating a small set of poisoned samples to aid the model's rapid unlearning of backdoor vulnerabilities. In the backdoor unlearning process, we present a novel token-based portion unlearning training regime.
  arXiv  Detail & Related papers  (2024-09-29T02:55:38Z)
• Weak-to-Strong Backdoor Attack for Large Language Models [15.055037707091435]
  We propose a novel backdoor attack algorithm from weak to strong based on feature alignment-enhanced knowledge distillation (W2SAttack) We demonstrate the superior performance of W2SAttack on classification tasks across four language models, four backdoor attack algorithms, and two different architectures of teacher models.
  arXiv  Detail & Related papers  (2024-09-26T15:20:37Z)
• Revisiting Backdoor Attacks against Large Vision-Language Models [76.42014292255944]
  This paper empirically examines the generalizability of backdoor attacks during the instruction tuning of LVLMs. We modify existing backdoor attacks based on the above key observations. This paper underscores that even simple traditional backdoor strategies pose a serious threat to LVLMs.
  arXiv  Detail & Related papers  (2024-06-27T02:31:03Z)
• SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks [53.28390057407576]
  Modern NLP models are often trained on public datasets drawn from diverse sources. Data poisoning attacks can manipulate the model's behavior in ways engineered by the attacker. Several strategies have been proposed to mitigate the risks associated with backdoor attacks.
  arXiv  Detail & Related papers  (2024-05-19T14:50:09Z)
• Attention-Enhancing Backdoor Attacks Against BERT-based Models [54.070555070629105]
  Investigating the strategies of backdoor attacks will help to understand the model's vulnerability. We propose a novel Trojan Attention Loss (TAL) which enhances the Trojan behavior by directly manipulating the attention patterns.
  arXiv  Detail & Related papers  (2023-10-23T01:24:56Z)
• Backdoor Attack with Sparse and Invisible Trigger [57.41876708712008]
  Deep neural networks (DNNs) are vulnerable to backdoor attacks. backdoor attack is an emerging yet threatening training-phase threat. We propose a sparse and invisible backdoor attack (SIBA)
  arXiv  Detail & Related papers  (2023-05-11T10:05:57Z)
• PiDAn: A Coherence Optimization Approach for Backdoor Attack Detection and Mitigation in Deep Neural Networks [22.900501880865658]
  Backdoor attacks impose a new threat in Deep Neural Networks (DNNs) We propose PiDAn, an algorithm based on coherence optimization purifying the poisoned data. Our PiDAn algorithm can detect more than 90% infected classes and identify 95% poisoned samples.
  arXiv  Detail & Related papers  (2022-03-17T12:37:21Z)
• DeepSweep: An Evaluation Framework for Mitigating DNN Backdoor Attacks using Data Augmentation [16.455154794893055]
  Third-party providers can inject poisoned samples into datasets or embed backdoors in Deep Learning models. We propose a systematic approach to discover the optimal policies for defending against different backdoor attacks. Our identified policy can effectively mitigate eight different kinds of backdoor attacks and outperform five existing defense methods.
  arXiv  Detail & Related papers  (2020-12-13T08:51:37Z)
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations [88.94162416324505]
  A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
  arXiv  Detail & Related papers  (2020-03-19T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.