Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information

• URL: http://arxiv.org/abs/2501.01544v1
• Date: Thu, 02 Jan 2025 21:31:38 GMT
• Title: Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
• Authors: Rasul Tutnov, Antoine Grosnit, Haitham Bou-Ammar,
• Abstract summary: Post-alignment of large language models (LLMs) is critical in improving their utility, safety, and alignment with human intentions. Direct preference optimisation (DPO) has become one of the most widely used algorithms for achieving this alignment. This paper introduces a unifying framework inspired by mutual information, which proposes a new loss function with flexible priors.
• Score: 5.655057078073446
• License:
• Abstract: Post-alignment of large language models (LLMs) is critical in improving their utility, safety, and alignment with human intentions. Direct preference optimisation (DPO) has become one of the most widely used algorithms for achieving this alignment, given its ability to optimise models based on human feedback directly. However, the vast number of DPO variants in the literature has made it increasingly difficult for researchers to navigate and fully grasp the connections between these approaches. This paper introduces a unifying framework inspired by mutual information, which proposes a new loss function with flexible priors. By carefully specifying these priors, we demonstrate that many existing algorithms, such as SimPO, TDPO, SparsePO, and others, can be derived from our framework. This unification offers a clearer and more structured approach, allowing researchers to understand the relationships between different DPO variants better. We aim to simplify the landscape of DPO algorithms, making it easier for the research community to gain insights and foster further advancements in LLM alignment. Ultimately, we hope our framework can be a foundation for developing more robust and interpretable alignment techniques.

Related papers

• Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment [74.25832963097658]
  Multi-Objective Alignment (MOA) aims to align responses with multiple human preference objectives. We find that DPO-based MOA approaches suffer from widespread preference conflicts in the data.
  arXiv  Detail & Related papers  (2025-02-20T08:27:00Z)
• SDPO: Segment-Level Direct Preference Optimization for Social Agents [56.970902914217156]
  Social agents powered by large language models (LLMs) can simulate human social behaviors but fall short in handling complex goal-oriented social dialogues. We propose Segment-Level Direct Preference Optimization (SDPO) to optimize multi-turn agent behavior while minimizing training noise.
  arXiv  Detail & Related papers  (2025-01-03T14:09:46Z)
• A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications [52.42860559005861]
  Direct Preference Optimization (DPO) has emerged as a promising approach for alignment. Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature.
  arXiv  Detail & Related papers  (2024-10-21T02:27:24Z)
• RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization [22.45649373554474]
  RainbowPO is a unified framework that categorizing key components into seven broad directions. We demonstrate that RainbowPO outperforms existing DPO variants. We provide insights to guide researchers in developing new DPO methods and assist practitioners in their implementations.
  arXiv  Detail & Related papers  (2024-10-05T15:44:46Z)
• Towards a Unified View of Preference Learning for Large Language Models: A Survey [88.66719962576005]
  Large Language Models (LLMs) exhibit remarkably powerful capabilities. One of the crucial factors to achieve success is aligning the LLM's output with human preferences. We decompose all the strategies in preference learning into four components: model, data, feedback, and algorithm.
  arXiv  Detail & Related papers  (2024-09-04T15:11:55Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. We increase the consistency and informativeness of the pairwise preference signals through targeted modifications. We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• The Hitchhiker's Guide to Human Alignment with *PO [43.4130314879284]
  We focus on identifying the algorithm that, while being performant, is simultaneously more robust to varying hyper parameters. Our analysis reveals that the widely adopted DPO method consistently produces lengthy responses of inferior quality. Motivated by these findings, we propose an embarrassingly simple extension to the DPO algorithm, LN-DPO, resulting in more concise responses without sacrificing quality.
  arXiv  Detail & Related papers  (2024-07-21T17:35:20Z)
• Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment [58.049113055986375]
  We develop a single stage approach named Alignment with Integrated Human Feedback (AIHF) to train reward models and the policy. The proposed approach admits a suite of efficient algorithms, which can easily reduce to, and leverage, popular alignment algorithms. We demonstrate the efficiency of the proposed solutions with extensive experiments involving alignment problems in LLMs and robotic control problems in MuJoCo.
  arXiv  Detail & Related papers  (2024-06-11T01:20:53Z)
• Surpassing legacy approaches to PWR core reload optimization with single-objective Reinforcement learning [0.0]
  We have developed methods based on Deep Reinforcement Learning (DRL) for both single- and multi-objective optimization. In this paper, we demonstrate the advantage of our RL-based approach, specifically using Proximal Policy Optimization (PPO) PPO adapts its search capability via a policy with learnable weights, allowing it to function as both a global and local search method.
  arXiv  Detail & Related papers  (2024-02-16T19:35:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.