Understanding Reference Policies in Direct Preference Optimization

• URL: http://arxiv.org/abs/2407.13709v2
• Date: Thu, 22 Aug 2024 17:56:15 GMT
• Title: Understanding Reference Policies in Direct Preference Optimization
• Authors: Yixin Liu, Pengfei Liu, Arman Cohan,
• Abstract summary: Direct Preference Optimization (DPO) has become a widely used training method for the instruction fine-tuning of large language models (LLMs) This work explores an under-investigated aspect of DPO - its dependency on the reference model or policy.
• Score: 50.67309013764383
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Direct Preference Optimization (DPO) has become a widely used training method for the instruction fine-tuning of large language models (LLMs). In this work, we explore an under-investigated aspect of DPO - its dependency on the reference model or policy. Such reference policies, typically instantiated as the model to be further fine-tuned, are important since they can impose an upper limit on DPO's effectiveness. Therefore, we address three related research questions in this work. First, we explore the optimal strength of the KL divergence constraint in DPO, which penalizes deviations from the reference policy, and find that DPO is sensitive to this strength. Next, we examine the necessity of the KL-constraint from the reference policies in DPO by providing both theoretical and empirical comparisons between DPO and related learning objectives, demonstrating DPO's superiority in this controlled setting. Additionally, we investigate whether DPO benefits from stronger reference policies, finding that a stronger reference policy can lead to improved performance, but only when it is similar to the model being fine-tuned. Our findings highlight the confounding role of reference policies in DPO and offer insights for best practices, while also identifying open research questions for future studies.

Related papers

• A Comprehensive Survey of Datasets, Theories, Variants, and Applications in Direct Preference Optimization [52.42860559005861]
  Direct Preference Optimization (DPO) has emerged as a promising approach for alignment. Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature.
  arXiv  Detail & Related papers  (2024-10-21T02:27:24Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. We increase the consistency and informativeness of the pairwise preference signals through targeted modifications. We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Reflective Policy Optimization [20.228281670899204]
  Reflective Policy Optimization (RPO) amalgamates past and future state-action information for policy optimization. RPO empowers the agent for introspection, allowing modifications to its actions within the current state. Empirical results demonstrate RPO's feasibility and efficacy in two reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2024-06-06T01:46:49Z)
• D2PO: Discriminator-Guided DPO with Response Evaluation Models [63.71853401569461]
  We propose D2PO, discriminator-guided DPO, for the online setting where preferences are being collected throughout learning. As we collect gold preferences, we use these not only to train our policy, but to train a discriminative response evaluation model to silver-label even more synthetic data for policy training. We show conditions under which silver labeling is most helpful: it is most effective when training the policy with DPO, outperforming traditional PPO, and benefits from maintaining a separate discriminator from the policy model.
  arXiv  Detail & Related papers  (2024-05-02T17:44:41Z)
• DPO Meets PPO: Reinforced Token Optimization for RLHF [36.97894955691627]
  We introduce a framework that models RLHF problems as a Markov decision process (MDP) Under this framework, we introduce an algorithm, dubbed as Reinforced Token Optimization (textttRTO), which learns the token-wise reward function from preference data. For its practical implementation, textttRTO innovatively integrates Direct Preference Optimization (DPO) and Proximal Policy Optimization.
  arXiv  Detail & Related papers  (2024-04-29T17:58:30Z)
• From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function [50.812404038684505]
  We show that we can derive DPO in the token-level MDP as a general inverse Q-learning algorithm, which satisfies the Bellman equation. We discuss applications of our work, including information elicitation in multi-turn dialogue, reasoning, agentic applications and end-to-end training of multi-model systems.
  arXiv  Detail & Related papers  (2024-04-18T17:37:02Z)
• Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective [25.34250859820326]
  We provide an analytical framework using the field theory to analyze the optimization process of DPO. We find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data.
  arXiv  Detail & Related papers  (2024-04-06T13:24:37Z)
• Fine-Tuning Language Models with Advantage-Induced Policy Alignment [80.96507425217472]
  We propose a novel algorithm for aligning large language models to human preferences. We show that it consistently outperforms PPO in language tasks by a large margin. We also provide a theoretical justification supporting the design of our loss function.
  arXiv  Detail & Related papers  (2023-06-04T01:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.