CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

• URL: http://arxiv.org/abs/2403.16649v2
• Date: Tue, 26 Mar 2024 06:08:20 GMT
• Title: CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment
• Authors: Feiteng Fang, Liang Zhu, Min Yang, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu,
• Abstract summary: Reinforcement learning from human feedback (RLHF) is a crucial technique in aligning large language models (LLMs) with human preferences. We present a simple yet effective Contrastive Learning Framework for Human Alignment (CLHA) to align LLMs with human preferences directly.
• Score: 42.71324708567498
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning from human feedback (RLHF) is a crucial technique in aligning large language models (LLMs) with human preferences, ensuring these LLMs behave in beneficial and comprehensible ways to users. However, a longstanding challenge in human alignment techniques based on reinforcement learning lies in their inherent complexity and difficulty in training. To address this challenge, we present a simple yet effective Contrastive Learning Framework for Human Alignment (CLHA) to align LLMs with human preferences directly. CLHA employs a novel rescoring strategy to evaluate the noise within the data by considering its inherent quality and dynamically adjusting the training process. Simultaneously, CLHA utilizes pairwise contrastive loss and adaptive supervised fine-tuning loss to adaptively modify the likelihood of generating responses, ensuring enhanced alignment with human preferences. Using advanced methods, CLHA surpasses other algorithms, showcasing superior performance in terms of reward model scores, automatic evaluations, and human assessments on the widely used ``Helpful and Harmless'' dataset.

Related papers

• RLTHF: Targeted Human Feedback for LLM Alignment [6.9866569384937325]
  Fine-tuning large language models to align with user preferences is challenging due to the high cost of quality human annotations. We propose RLTHF, a human-AI hybrid framework that combines LLM-based initial alignment with selective human annotations. We show that RLTHF reaches full-human annotation-level alignment with only 6-7% of the human annotation effort.
  arXiv  Detail & Related papers  (2025-02-19T04:25:11Z)
• Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models [94.39278422567955]
  Fine-tuning large language models (LLMs) on human preferences has proven successful in enhancing their capabilities. However, ensuring the safety of LLMs during the fine-tuning remains a critical concern. We propose a supervised learning framework called Bi-Factorial Preference Optimization (BFPO) to address this issue.
  arXiv  Detail & Related papers  (2024-08-27T17:31:21Z)
• Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment [58.049113055986375]
  We develop a single stage approach named Alignment with Integrated Human Feedback (AIHF) to train reward models and the policy. The proposed approach admits a suite of efficient algorithms, which can easily reduce to, and leverage, popular alignment algorithms. We demonstrate the efficiency of the proposed solutions with extensive experiments involving alignment problems in LLMs and robotic control problems in MuJoCo.
  arXiv  Detail & Related papers  (2024-06-11T01:20:53Z)
• RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs [49.386699863989335]
  Training large language models (LLMs) to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals.
  arXiv  Detail & Related papers  (2024-04-12T15:54:15Z)
• COPR: Continual Human Preference Learning via Optimal Policy Regularization [54.4973136224034]
  Reinforcement Learning from Human Feedback (RLHF) is commonly utilized to improve the alignment of Large Language Models (LLMs) with human preferences. We propose the Continual Optimal Policy Regularization (COPR) method, which draws inspiration from the optimal policy theory.
  arXiv  Detail & Related papers  (2024-02-22T02:20:08Z)
• Aligning Large Language Models with Human Preferences through Representation Engineering [41.81020951061438]
  Drawing inspiration from the emerging field of representation engineering (RepE), this study aims to identify relevant representations for high-level human preferences embedded in patterns of activity within an LLM. This novel approach, denoted as Representation Alignment from Human Feedback (RAHF), proves to be effective, computationally efficient, and easy to implement.
  arXiv  Detail & Related papers  (2023-12-26T11:01:36Z)
• COPR: Continual Learning Human Preference through Optimal Policy Regularization [32.54658750353585]
  We propose a new method called Continual Optimal Policy Regularization (COPR) COPR involves a single learning phase and doesn't necessitate complex reinforcement learning. Our experimental results show that COPR outperforms strong Continuous Learning (CL) baselines.
  arXiv  Detail & Related papers  (2023-10-24T10:05:32Z)
• SALMON: Self-Alignment with Instructable Reward Models [80.83323636730341]
  This paper presents a novel approach, namely SALMON, to align base language models with minimal human supervision. We develop an AI assistant named Dromedary-2 with only 6 exemplars for in-context learning and 31 human-defined principles.
  arXiv  Detail & Related papers  (2023-10-09T17:56:53Z)
• SLiC-HF: Sequence Likelihood Calibration with Human Feedback [35.74135968442311]
  We show how the recently introduced Sequence Likelihood (SLiC) can also be used to effectively learn from human preferences. Experiments on the TL;DR summarization task show that SLiC-HF significantly improves supervised fine-tuning baselines.
  arXiv  Detail & Related papers  (2023-05-17T17:57:10Z)
• Fine-tuning Language Models with Generative Adversarial Reward Modelling [30.424363135421917]
  Reinforcement Learning with Human Feedback (RLHF) has been demonstrated to significantly enhance the performance of large language models (LLMs) We propose another alternative approach: Reinforcement Learning with Generative Adversarial Feedback (RLGAF) to RLHF and SFT.
  arXiv  Detail & Related papers  (2023-05-09T17:06:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.