Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

• URL: http://arxiv.org/abs/2404.05530v2
• Date: Tue, 6 Aug 2024 14:30:31 GMT
• Title: Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
• Authors: Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler,
• Abstract summary: Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training. We study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences.
• Score: 30.343186069189944
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: injecting a small amount of poisonous data (1-5\% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.

Related papers

• Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF [67.48004037550064]
  We propose an active learning approach to efficiently select prompt and preference pairs. Our method evaluates the gradients of all potential preference annotations to assess their impact on model updates. Experimental results demonstrate that our method outperforms the baseline by up to 5% in win rates against the chosen completion.
  arXiv  Detail & Related papers  (2025-03-28T04:22:53Z)
• Policy Teaching via Data Poisoning in Learning from Human Preferences [24.645259298082436]
  We study data poisoning attacks in learning from human preferences. We study the problem of teaching/enforcing a target policy $pidagger$ by synthesizing preference data.
  arXiv  Detail & Related papers  (2025-03-13T10:11:54Z)
• PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning [32.508939142492004]
  We introduce PoisonBench, a benchmark for evaluating large language models' susceptibility to data poisoning during preference learning. Data poisoning attacks can manipulate large language model responses to include hidden malicious content or biases. We deploy two distinct attack types across eight realistic scenarios, assessing 21 widely-used models.
  arXiv  Detail & Related papers  (2024-10-11T13:50:50Z)
• Unlearnable Examples Detection via Iterative Filtering [84.59070204221366]
  Deep neural networks are proven to be vulnerable to data poisoning attacks. It is quite beneficial and challenging to detect poisoned samples from a mixed dataset. We propose an Iterative Filtering approach for UEs identification.
  arXiv  Detail & Related papers  (2024-08-15T13:26:13Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• Progressive Poisoned Data Isolation for Training-time Backdoor Defense [23.955347169187917]
  Deep Neural Networks (DNN) are susceptible to backdoor attacks where malicious attackers manipulate the model's predictions via data poisoning. In this study, we present a novel and efficacious defense method, termed Progressive Isolation of Poisoned Data (PIPD) Our PIPD achieves an average True Positive Rate (TPR) of 99.95% and an average False Positive Rate (FPR) of 0.06% for diverse attacks over CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2023-12-20T02:40:28Z)
• RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models [62.72318564072706]
  Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences. Despite its advantages, RLHF relies on human annotators to rank the text. We propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors.
  arXiv  Detail & Related papers  (2023-11-16T07:48:45Z)
• Explore the Effect of Data Selection on Poison Efficiency in Backdoor Attacks [10.817607451423765]
  In this study, we focus on improving the poisoning efficiency of backdoor attacks from the sample selection perspective. We adopt the forgetting events of the samples to indicate the contribution of different poisoned samples and use the curvature of the loss surface to analyses the effectiveness of this phenomenon.
  arXiv  Detail & Related papers  (2023-10-15T05:55:23Z)
• On Practical Aspects of Aggregation Defenses against Data Poisoning Attacks [58.718697580177356]
  Attacks on deep learning models with malicious training samples are known as data poisoning. Recent advances in defense strategies against data poisoning have highlighted the effectiveness of aggregation schemes in achieving certified poisoning robustness. Here we focus on Deep Partition Aggregation, a representative aggregation defense, and assess its practical aspects, including efficiency, performance, and robustness.
  arXiv  Detail & Related papers  (2023-06-28T17:59:35Z)
• Data-Efficient Backdoor Attacks [14.230326737098554]
  Deep neural networks are vulnerable to backdoor attacks. In this paper, we formulate improving the poisoned data efficiency by the selection. The same attack success rate can be achieved with only 47% to 75% of the poisoned sample volume.
  arXiv  Detail & Related papers  (2022-04-22T09:52:22Z)
• Accumulative Poisoning Attacks on Real-time Data [56.96241557830253]
  We show that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects. Our work validates that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects.
  arXiv  Detail & Related papers  (2021-06-18T08:29:53Z)
• Property Inference From Poisoning [15.105224455937025]
  Property inference attacks consider an adversary who has access to the trained model and tries to extract some global statistics of the training data. We study poisoning attacks where the goal of the adversary is to increase the information leakage of the model. Our findings suggest that poisoning attacks can boost the information leakage significantly and should be considered as a stronger threat model in sensitive applications.
  arXiv  Detail & Related papers  (2021-01-26T20:35:28Z)
• How Robust are Randomized Smoothing based Defenses to Data Poisoning? [66.80663779176979]
  We present a previously unrecognized threat to robust machine learning models that highlights the importance of training-data quality. We propose a novel bilevel optimization-based data poisoning attack that degrades the robustness guarantees of certifiably robust classifiers. Our attack is effective even when the victim trains the models from scratch using state-of-the-art robust training methods.
  arXiv  Detail & Related papers  (2020-12-02T15:30:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.