Mitigating the Alignment Tax of RLHF
- URL: http://arxiv.org/abs/2309.06256v3
- Date: Mon, 5 Feb 2024 06:43:17 GMT
- Title: Mitigating the Alignment Tax of RLHF
- Authors: Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng
Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie
Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang
- Abstract summary: Reinforcement Learning with Human Feedback (RLHF) can lead to, which is also known as the alignment tax.
We propose model averaging, which interpolates between pre and post RLHF model weights, to achieve a more efficient reward-tax front.
- Score: 77.7879015461373
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: LLMs acquire a wide range of abilities during pre-training, but aligning LLMs
under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting,
which is also known as the alignment tax. To empirically verify this
hypothesis, we conducted experiments with existing RLHF algorithms using
OpenLLaMA-3B, which revealed a pronounced alignment tax in NLP tasks. On the
other hand, despite various techniques to mitigate forgetting, they are often
at odds with the RLHF performance, leading to a trade-off between reward
maximization and forgetting mitigation.
In light of the above pressing issue in aligning LLMs, in this paper we
explore model averaging, which interpolates between pre and post RLHF model
weights, to achieve a more efficient reward-tax Pareto front. To understand its
effectiveness, We offer theoretical insights into model averaging, revealing
that it enhances performance Pareto front by increasing feature diversity on
the layers where tasks share overlapped feature spaces. Empirical evidence
corroborates our analysis by showing the benefits of averaging low-level
transformer layers. Building on the analysis and the observation that averaging
different layers of the transformer leads to significantly different reward-tax
trade-offs, we propose Adaptive Model Averaging (AMA) to adaptively find
various combination ratios of model layers. AMA seeks to maximize the alignment
reward while incurring minimal alignment tax. Moreover, we validate AMA's
performance across a range of RLHF algorithms over OpenLLaMA-3B and further
extend our findings to Mistral-7B.
Related papers
- Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment [47.682736928029996]
Large Language Models (LLMs) are designed to align with human-centric values while preventing the degradation of abilities acquired through Pre-training and Supervised Fine-tuning (SFT)
In this paper, we show that interpolating RLHF and SFT model parameters can adjust the trade-off between human preference and basic capabilities, thereby reducing the alignment tax.
It significantly enhances alignment reward while mitigating alignment tax, achieving higher overall performance across 14 benchmarks.
arXiv Detail & Related papers (2024-05-28T07:53:40Z) - Teaching Large Language Models to Reason with Reinforcement Learning [38.17625148525193]
Reinforcement Learning from Human Feedback (textbfRLHF) has emerged as a dominant approach for aligning LLM outputs with human preferences.
Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback.
arXiv Detail & Related papers (2024-03-07T16:36:29Z) - Uncertainty-Penalized Reinforcement Learning from Human Feedback with
Diverse Reward LoRA Ensembles [26.955375398765085]
Reinforcement learning from human feedback (RLHF) emerges as a promising paradigm for aligning large language models (LLMs)
In this paper, we observe the weakness of KL regularization which is commonly employed in existing RLHF methods to address overoptimization.
We propose uncertainty-penalized RLHF (UP-RLHF), which incorporates uncertainty regularization during RL-finetuning.
arXiv Detail & Related papers (2023-12-30T14:14:14Z) - On Diversified Preferences of Large Language Model Alignment [51.26149027399505]
We investigate the impact of diversified preferences on reward modeling.
We find that diversified preference data negatively affect the calibration performance of reward models.
We propose a novel Multi-Objective Reward learning method to enhance the calibration performance of RMs on shared preferences.
arXiv Detail & Related papers (2023-12-12T16:17:15Z) - Routing to the Expert: Efficient Reward-guided Ensemble of Large
Language Models [69.51130760097818]
We propose Zooter, a reward-guided routing method distilling rewards on training queries to train a routing function.
We evaluate Zooter on a comprehensive benchmark collection with 26 subsets on different domains and tasks.
arXiv Detail & Related papers (2023-11-15T04:40:43Z) - Understanding the Effects of RLHF on LLM Generalisation and Diversity [26.56388427640671]
Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) have been used in some of the most widely deployed AI models to date.
We present an analysis of how each stage of the process affects two key properties: out-of-distribution (OOD) generalisation and output diversity.
arXiv Detail & Related papers (2023-10-10T09:25:44Z) - Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity [88.62935593360162]
Large Language Models (LLMs) are renowned for their remarkable performance across diverse domains.
We introduce a novel LLM pruning methodology that incorporates a tailored set of non-uniform layerwise sparsity ratios, termed as Outlier Weighed Layerwise sparsity (OWL)
OWL exhibits a remarkable performance gain, surpassing the state-of-the-art Wanda and SparseGPT by 61.22 and 6.80 perplexity at a high sparsity level of 70%, respectively.
arXiv Detail & Related papers (2023-10-08T14:22:58Z) - Principled Reinforcement Learning with Human Feedback from Pairwise or
$K$-wise Comparisons [79.98542868281473]
We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF)
We show that when training a policy based on the learned reward model, MLE fails while a pessimistic MLE provides policies with improved performance under certain coverage assumptions.
arXiv Detail & Related papers (2023-01-26T18:07:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.