The Limits of Preference Data for Post-Training

• URL: http://arxiv.org/abs/2505.19964v1
• Date: Mon, 26 May 2025 13:26:15 GMT
• Title: The Limits of Preference Data for Post-Training
• Authors: Eric Zhao, Jessica Dai, Pranjal Awasthi,
• Abstract summary: We show that preference data fundamentally and significantly limits outcome-based optimization.<n>We formalize this impossibility using voting theory, drawing an analogy between how a model chooses to answer a query with how voters choose a candidate to elect.<n>This indicates that grounded human scoring and algorithmic innovations are necessary for extending the success of RL post-training to domains demanding human feedback.
• Score: 27.229909368242517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in strengthening the capabilities of large language models has stemmed from applying reinforcement learning to domains with automatically verifiable outcomes. A key question is whether we can similarly use RL to optimize for outcomes in domains where evaluating outcomes inherently requires human feedback; for example, in tasks like deep research and trip planning, outcome evaluation is qualitative and there are many possible degrees of success. One attractive and scalable modality for collecting human feedback is preference data: ordinal rankings (pairwise or $k$-wise) that indicate, for $k$ given outcomes, which one is preferred. In this work, we study a critical roadblock: preference data fundamentally and significantly limits outcome-based optimization. Even with idealized preference data (infinite, noiseless, and online), the use of ordinal feedback can prevent obtaining even approximately optimal solutions. We formalize this impossibility using voting theory, drawing an analogy between how a model chooses to answer a query with how voters choose a candidate to elect. This indicates that grounded human scoring and algorithmic innovations are necessary for extending the success of RL post-training to domains demanding human feedback. We also explore why these limitations have disproportionately impacted RLHF when it comes to eliciting reasoning behaviors (e.g., backtracking) versus situations where RLHF has been historically successful (e.g., instruction-tuning and safety training), finding that the limitations of preference data primarily suppress RLHF's ability to elicit robust strategies -- a class that encompasses most reasoning behaviors.

Related papers

• Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback [0.0]
  LLMs rely on optimizing preference-based objectives, where these preferences are typically elicited as ordinal, binary choices between responses.<n>Recent work has focused on improving label quality or mitigating particular biases, but we identify a more fundamental limitation: these methods collect the wrong kind of data.<n>We show that selecting the optimal model requires recovering preferences over emphmodels (rather than just responses), which can only be identified given cardinal feedback about response quality.
  arXiv  Detail & Related papers  (2025-08-11T21:42:33Z)
• Behavior Preference Regression for Offline Reinforcement Learning [0.0]
  offline reinforcement learning (RL) methods aim to learn optimal policies with access only to trajectories in a fixed dataset.<n>Policy constraint methods formulate policy learning as an optimization problem that balances maximizing reward with minimizing deviation from the policy.<n>We adapt this approach of paired comparison to Behavior Regression Preference for offline RL.<n>We empirically evaluate BPR on the widely used D4RL Locomotion and Antmaze datasets, as well as the more challenging V-D4RL suite.
  arXiv  Detail & Related papers  (2025-03-02T15:13:02Z)
• PILAF: Optimal Human Preference Sampling for Reward Modeling [14.336058926701432]
  We propose Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response sampling strategy for preference labeling.<n>PILAF explicitly aligns preference learning with maximizing the underlying oracle reward.
  arXiv  Detail & Related papers  (2025-02-06T18:09:00Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We show that our approach consistently boosts DPO by a considerable margin.<n>Our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• Online Bandit Learning with Offline Preference Data for Improved RLHF [15.799929216215672]
  We propose a posterior sampling algorithm for online learning that can be warm-started with an offline dataset with noisy preference feedback.<n>We show that by modeling the 'competence' of the expert that generated it, we are able to use such a dataset most effectively.
  arXiv  Detail & Related papers  (2024-06-13T20:25:52Z)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data [102.16105233826917]
  Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning.
  arXiv  Detail & Related papers  (2024-04-22T17:20:18Z)
• Sample Efficient Preference Alignment in LLMs via Active Exploration [63.84454768573154]
  We take advantage of the fact that one can often choose contexts at which to obtain human feedback to most efficiently identify a good policy.<n>We propose an active exploration algorithm to efficiently select the data and provide theoretical proof that it has a worst-case regret bound.<n>Our method outperforms the baselines with limited samples of human preferences on several language models and four real-world datasets.
  arXiv  Detail & Related papers  (2023-12-01T00:54:02Z)
• Contrastive Preference Learning: Learning from Human Feedback without RL [71.77024922527642]
  We introduce Contrastive Preference Learning (CPL), an algorithm for learning optimal policies from preferences without learning reward functions. CPL is fully off-policy, uses only a simple contrastive objective, and can be applied to arbitrary MDPs.
  arXiv  Detail & Related papers  (2023-10-20T16:37:56Z)
• Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networks [58.469818546042696]
  We study the sample efficiency of OPE with human preference and establish a statistical guarantee for it. By appropriately selecting the size of a ReLU network, we show that one can leverage any low-dimensional manifold structure in the Markov decision process.
  arXiv  Detail & Related papers  (2023-10-16T16:27:06Z)
• Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism [91.52263068880484]
  We study offline Reinforcement Learning with Human Feedback (RLHF) We aim to learn the human's underlying reward and the MDP's optimal policy from a set of trajectories induced by human choices. RLHF is challenging for multiple reasons: large state space but limited human feedback, the bounded rationality of human decisions, and the off-policy distribution shift.
  arXiv  Detail & Related papers  (2023-05-29T01:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.