DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints

• URL: http://arxiv.org/abs/2405.19026v2
• Date: Fri, 20 Dec 2024 07:37:32 GMT
• Title: DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints
• Authors: Andrew Zhao, Quentin Xu, Matthieu Lin, Shenzhi Wang, Yong-jin Liu, Zilong Zheng, Gao Huang,
• Abstract summary: We introduce DiveR-CT, which relaxes conventional constraints on the objective and semantic reward, granting greater freedom for the policy to enhance diversity. Our experiments demonstrate DiveR-CT's marked superiority over baselines by 1) generating data that perform better in various diversity metrics across different attack success rate levels, 2) better-enhancing resiliency in blue team models through safety tuning based on collected data, 3) allowing dynamic control of objective weights for reliable and controllable attack success rates, and 4) reducing susceptibility to reward overoptimization.
• Score: 68.82294911302579
• License:
• Abstract: Recent advances in large language model assistants have made them indispensable, raising significant concerns over managing their safety. Automated red teaming offers a promising alternative to the labor-intensive and error-prone manual probing for vulnerabilities, providing more consistent and scalable safety evaluations. However, existing approaches often compromise diversity by focusing on maximizing attack success rate. Additionally, methods that decrease the cosine similarity from historical embeddings with semantic diversity rewards lead to novelty stagnation as history grows. To address these issues, we introduce DiveR-CT, which relaxes conventional constraints on the objective and semantic reward, granting greater freedom for the policy to enhance diversity. Our experiments demonstrate DiveR-CT's marked superiority over baselines by 1) generating data that perform better in various diversity metrics across different attack success rate levels, 2) better-enhancing resiliency in blue team models through safety tuning based on collected data, 3) allowing dynamic control of objective weights for reliable and controllable attack success rates, and 4) reducing susceptibility to reward overoptimization. Overall, our method provides an effective and efficient approach to LLM red teaming, accelerating real-world deployment.

Related papers

• Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction [24.499874512829198]
  We proposeHolistic Automated Red teaMing, which scales up the diversity of test cases based on an adversarial, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn probing in a human-like manner.
  arXiv  Detail & Related papers  (2024-09-25T09:44:48Z)
• SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Models [19.486685336959482]
  Large language models (LLMs) continue to advance in capability and influence, ensuring their security and preventing harmful outputs has become crucial. A promising approach to address these concerns involves training models to automatically generate adversarial prompts for red teaming. We introduce the $mathbfStextelf-mathbfEtextvolving mathbfAtextdversarial mathbfStextafetyety mathbf(SEAS)$ optimization framework, which enhances security by leveraging data generated by the model itself.
  arXiv  Detail & Related papers  (2024-08-05T16:55:06Z)
• Learning diverse attacks on large language models for robust red-teaming and safety tuning [126.32539952157083]
  Red-teaming, or identifying prompts that elicit harmful responses, is a critical step in ensuring the safe deployment of large language models. We show that even with explicit regularization to favor novelty and diversity, existing approaches suffer from mode collapse or fail to generate effective attacks. We propose to use GFlowNet fine-tuning, followed by a secondary smoothing phase, to train the attacker model to generate diverse and effective attack prompts.
  arXiv  Detail & Related papers  (2024-05-28T19:16:17Z)
• Ensemble Adversarial Defense via Integration of Multiple Dispersed Low Curvature Models [7.8245455684263545]
  In this work, we aim to enhance ensemble diversity by reducing attack transferability. We identify second-order gradients, which depict the loss curvature, as a key factor in adversarial robustness. We introduce a novel regularizer to train multiple more-diverse low-curvature network models.
  arXiv  Detail & Related papers  (2024-03-25T03:44:36Z)
• RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content [62.685566387625975]
  Current mitigation strategies, while effective, are not resilient under adversarial attacks. This paper introduces Resilient Guardrails for Large Language Models (RigorLLM), a novel framework designed to efficiently moderate harmful and unsafe inputs.
  arXiv  Detail & Related papers  (2024-03-19T07:25:02Z)
• Improving Generalization of Alignment with Human Preferences through Group Invariant Learning [56.19242260613749]
  Reinforcement Learning from Human Feedback (RLHF) enables the generation of responses more aligned with human preferences. Previous work shows that Reinforcement Learning (RL) often exploits shortcuts to attain high rewards and overlooks challenging samples. We propose a novel approach that can learn a consistent policy via RL across various data groups or domains.
  arXiv  Detail & Related papers  (2023-10-18T13:54:15Z)
• Making Substitute Models More Bayesian Can Enhance Transferability of Adversarial Examples [89.85593878754571]
  transferability of adversarial examples across deep neural networks is the crux of many black-box attacks. We advocate to attack a Bayesian model for achieving desirable transferability. Our method outperforms recent state-of-the-arts by large margins.
  arXiv  Detail & Related papers  (2023-02-10T07:08:13Z)
• Choosing the Best of Both Worlds: Diverse and Novel Recommendations through Multi-Objective Reinforcement Learning [68.45370492516531]
  We introduce Scalarized Multi-Objective Reinforcement Learning (SMORL) for the Recommender Systems (RS) setting. SMORL agent augments standard recommendation models with additional RL layers that enforce it to simultaneously satisfy three principal objectives: accuracy, diversity, and novelty of recommendations. Our experimental results on two real-world datasets reveal a substantial increase in aggregate diversity, a moderate increase in accuracy, reduced repetitiveness of recommendations, and demonstrate the importance of reinforcing diversity and novelty as complementary objectives.
  arXiv  Detail & Related papers  (2021-10-28T13:22:45Z)
• DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles [20.46399318111058]
  Adversarial attacks can mislead CNN models with small perturbations, which can effectively transfer between different models trained on the same dataset. We propose DVERGE, which isolates the adversarial vulnerability in each sub-model by distilling non-robust features. The novel diversity metric and training procedure enables DVERGE to achieve higher robustness against transfer attacks.
  arXiv  Detail & Related papers  (2020-09-30T14:57:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.