Decoupled Alignment for Robust Plug-and-Play Adaptation

• URL: http://arxiv.org/abs/2406.01514v3
• Date: Thu, 6 Jun 2024 04:25:40 GMT
• Title: Decoupled Alignment for Robust Plug-and-Play Adaptation
• Authors: Haozheng Luo, Jiahao Yu, Wenxin Zhang, Jialong Li, Jerry Yao-Chieh Hu, Xinyu Xing, Han Liu,
• Abstract summary: We introduce a low-resource safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF) Our main idea is to exploit knowledge distillation to extract the alignment information from existing well-aligned LLMs and integrate it into unaligned LLMs in a plug-and-play fashion. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.41%, reaching as high as 51.39%, in 17 unaligned pre-trained LLMs, without compromising performance.
• Score: 19.10463167105986
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a low-resource safety enhancement method for aligning large language models (LLMs) without the need for supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF). Our main idea is to exploit knowledge distillation to extract the alignment information from existing well-aligned LLMs and integrate it into unaligned LLMs in a plug-and-play fashion. Methodology, we employ delta debugging to identify the critical components of knowledge necessary for effective distillation. On the harmful question dataset, our method significantly enhances the average defense success rate by approximately 14.41%, reaching as high as 51.39%, in 17 unaligned pre-trained LLMs, without compromising performance.

Related papers

• RIDE: Enhancing Large Language Model Alignment through Restyled In-Context Learning Demonstration Exemplars [57.6513924960128]
  Alignment tuning is crucial for ensuring large language models (LLMs) behave ethically and helpfully. This paper proposes a low-cost, tuning-free method using in-context learning (ICL) to enhance LLM alignment.
  arXiv  Detail & Related papers  (2025-02-17T11:16:19Z)
• Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models [24.168387024091082]
  Fine-tuning large language models (LLMs) based on human preferences has been effective in improving their performance. Maintaining safety throughout the fine-tuning process remains a significant challenge. We propose an Equilibrate RLHF framework that achieves better safety alignment even with fewer training data.
  arXiv  Detail & Related papers  (2025-02-17T08:40:30Z)
• SPARC: Subspace-Aware Prompt Adaptation for Robust Continual Learning in LLMs [4.194295877935867]
  We propose a lightweight continual learning framework for large language models (LLMs) Our method achieves high knowledge retention in both task-incremental and domain-incremental continual learning setups. Experiments on the SuperGLUE benchmark demonstrate that our PCA-based prompt tuning combined with LoRA maintains full knowledge retention while improving accuracy, utilizing only 1% of the model's parameters.
  arXiv  Detail & Related papers  (2025-02-05T06:11:55Z)
• Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities [63.603861880022954]
  We introduce ADV-LLM, an iterative self-tuning process that crafts adversarial LLMs with enhanced jailbreak ability. Our framework significantly reduces the computational cost of generating adversarial suffixes while achieving nearly 100% ASR on various open-source LLMs. It exhibits strong attack transferability to closed-source models, achieving 99% ASR on GPT-3.5 and 49% ASR on GPT-4, despite being optimized solely on Llama3.
  arXiv  Detail & Related papers  (2024-10-24T06:36:12Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models [94.39278422567955]
  Fine-tuning large language models (LLMs) on human preferences has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during the fine-tuning remains a critical concern. We propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO) to address this issue.
  arXiv  Detail & Related papers  (2024-08-27T17:31:21Z)
• DistillSeq: A Framework for Safety Alignment Testing in Large Language Models using Knowledge Distillation [4.340880264464675]
  Large Language Models (LLMs) have showcased their remarkable capabilities in diverse domains, encompassing natural language understanding, translation, and even code generation. The potential for LLMs to generate harmful content is a significant concern. This study explores cost-saving strategies during the testing phase to balance the need for thorough evaluation with the constraints of resource availability.
  arXiv  Detail & Related papers  (2024-07-14T07:21:54Z)
• PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference [9.883296844539839]
  The PKU-SafeRLHF dataset is designed to promote research on safety alignment in large language models (LLMs) Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe.
  arXiv  Detail & Related papers  (2024-06-20T18:37:36Z)
• ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding [89.0074567748505]
  We present reverse prompt contrastive decoding (ROSE), a simple-yet-effective method to boost the safety of existing instruction-tuned LLMs without any additional training. Experiments on 6 safety and 2 general-purpose tasks show that, our ROSE not only brings consistent and significant safety improvements (up to +13.8% safety score) upon 5 types of instruction-tuned LLMs, but also benefits the general-purpose ability of LLMs.
  arXiv  Detail & Related papers  (2024-02-19T06:58:42Z)
• Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data [100.33096338195723]
  We focus on Few-shot Learning with Auxiliary Data (FLAD) FLAD assumes access to auxiliary data during few-shot learning in hopes of improving generalization. We propose two algorithms -- EXP3-FLAD and UCB1-FLAD -- and compare them with prior FLAD methods that either explore or exploit.
  arXiv  Detail & Related papers  (2023-02-01T18:59:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.