Selective Preference Optimization via Token-Level Reward Function Estimation

• URL: http://arxiv.org/abs/2408.13518v1
• Date: Sat, 24 Aug 2024 08:44:04 GMT
• Title: Selective Preference Optimization via Token-Level Reward Function Estimation
• Authors: Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, Sophia Ananiadou,
• Abstract summary: We propose Selective Preference Optimization (SePO), a novel selective alignment strategy that centers on efficient key token selection. SePO applies to any existing alignment datasets with response-level annotations. Experiments show that SePO significantly outperforms competitive baseline methods by only optimizing 30% key tokens on the target dataset.
• Score: 34.575466253492436
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in large language model alignment leverage token-level supervisions to perform fine-grained preference optimization. However, existing token-level alignment methods either optimize on all available tokens, which can be noisy and inefficient, or perform selective training with complex and expensive key token selection strategies. In this work, we propose Selective Preference Optimization (SePO), a novel selective alignment strategy that centers on efficient key token selection. SePO proposes the first token selection method based on Direct Preference Optimization (DPO), which trains an oracle model to estimate a token-level reward function on the target data. This method applies to any existing alignment datasets with response-level annotations and enables cost-efficient token selection with small-scale oracle models and training data. The estimated reward function is then utilized to score all tokens within the target dataset, where only the key tokens are selected to supervise the target policy model with a reference model-free contrastive objective function. Extensive experiments on three public evaluation benchmarks show that SePO significantly outperforms competitive baseline methods by only optimizing 30% key tokens on the target dataset. SePO applications on weak-to-strong generalization show that weak oracle models effectively supervise strong policy models with up to 16.8x more parameters. SePO also effectively selects key tokens from out-of-distribution data to enhance strong policy models and alleviate the over-optimization problem.

Related papers

• Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization [0.0]
  Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance.<n>This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs.<n>By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity.
  arXiv  Detail & Related papers  (2025-07-10T12:58:45Z)
• ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization [48.50761200321113]
  We introduce ConfPO, a method for preference learning in Large Language Models (LLMs)<n>It identifies and optimize preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute.<n> Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs.
  arXiv  Detail & Related papers  (2025-06-10T11:54:22Z)
• Adaptive Sample Scheduling for Direct Preference Optimization [37.75208455935495]
  We introduce a novel problem: Sample Scheduling for DPO.<n>It aims to dynamically and adaptively schedule training samples based on the model's evolving states.<n>We propose SamS, an efficient and effective algorithm that adaptively selects samples in each training batch.
  arXiv  Detail & Related papers  (2025-06-08T10:26:09Z)
• ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment [94.36403843133616]
  Using human preferences to align large language models (LLMs) has significantly improved their performance in various downstream tasks.<n>Existing methods either lack a strong theoretical foundation or depend on restrictive reward function assumptions.<n>We propose an algorithm, ActiveDPO, that uses a theoretically grounded data selection criterion for non-linear reward functions.
  arXiv  Detail & Related papers  (2025-05-25T17:42:52Z)
• Review, Refine, Repeat: Understanding Iterative Decoding of AI Agents with Dynamic Evaluation and Selection [71.92083784393418]
  Inference-time methods such as Best-of-N (BON) sampling offer a simple yet effective alternative to improve performance. We propose Iterative Agent Decoding (IAD) which combines iterative refinement with dynamic candidate evaluation and selection guided by a verifier.
  arXiv  Detail & Related papers  (2025-04-02T17:40:47Z)
• $α$-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs [45.46582930202524]
  $alpha$-DPO is an adaptive preference optimization algorithm for large language models. It balances the policy model and the reference model to achieve personalized reward margins. It consistently outperforms DPO and SimPO across various model settings.
  arXiv  Detail & Related papers  (2024-10-14T04:29:57Z)
• TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights [73.9088920210495]
  We propose a token-level importance sampling DPO objective named TIS-DPO that assigns importance weights to each token based on its reward. TIS-DPO significantly outperforms various baseline methods on harmlessness and helpfulness alignment and summarization tasks.
  arXiv  Detail & Related papers  (2024-10-06T04:03:00Z)
• An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting [53.36437745983783]
  We first construct a max-margin optimization-based model to model potentially non-monotonic preferences. We devise information amount measurement methods and question selection strategies to pinpoint the most informative alternative in each iteration. Two incremental preference elicitation-based algorithms are developed to learn potentially non-monotonic preferences.
  arXiv  Detail & Related papers  (2024-09-04T14:36:20Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC. We increase the consistency and informativeness of the pairwise preference signals through targeted modifications. We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• Pareto-Optimal Learning from Preferences with Hidden Context [17.590330740964266]
  We propose POPL, which enables pluralistic alignment by framing discrepant group preferences as objectives with potential trade-offs. Our theoretical and empirical evaluations demonstrate that POPL surpasses baseline methods in learning sets of reward functions and policies. We illustrate that POPL can also serve as a foundation for techniques optimizing specific notions of group fairness.
  arXiv  Detail & Related papers  (2024-06-21T18:57:38Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• DPO Meets PPO: Reinforced Token Optimization for RLHF [35.638723885233475]
  We introduce an algorithm that learns the token-wise reward function from preference data and performs policy optimization based on this learned token-wise reward signal. Experiments demonstrate that textttRTO performs better than PPO and other direct preference learning algorithms.
  arXiv  Detail & Related papers  (2024-04-29T17:58:30Z)
• Token-level Direct Preference Optimization [8.249403373337024]
  Fine-tuning pre-trained Large Language Models is essential to align them with human values and intentions. We introduce Token-level Direct Preference Optimization (TDPO), a novel approach to align LLMs with human preferences by optimizing policy at the token level.
  arXiv  Detail & Related papers  (2024-04-18T08:49:38Z)
• Towards Efficient Exact Optimization of Language Model Alignment [93.39181634597877]
  Direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. We show that DPO derived based on the optimal solution of problem leads to a compromised mean-seeking approximation of the optimal solution in practice. We propose efficient exact optimization (EXO) of the alignment objective.
  arXiv  Detail & Related papers  (2024-02-01T18:51:54Z)
• S$^3$VAADA: Submodular Subset Selection for Virtual Adversarial Active Domain Adaptation [49.01925978752677]
  In the real-world scenario's it might be feasible to get labels for a small proportion of target data. We propose S$3$VAADA which i) introduces a novel submodular criterion to select a maximally informative subset to label and ii) enhances a cluster-based DA procedure. Our approach consistently outperforms the competing state-of-the-art approaches on datasets with varying degrees of domain shifts.
  arXiv  Detail & Related papers  (2021-09-18T10:53:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.