From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning

• URL: http://arxiv.org/abs/2606.01123v1
• Date: Sun, 31 May 2026 09:46:53 GMT
• Title: From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
• Abstract summary: Preference-based reinforcement learning avoids explicit reward engineering.<n>We propose a new training framework that learns latent successor-measure representations from reward-free offline data.<n>We show that our method achieves superior preference efficiency over offline PbRL baselines.
• Score: 19.663565323513577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Preference-based reinforcement learning (PbRL) avoids explicit reward engineering by learning from pairwise human preference feedback. Existing offline PbRL methods typically follow a two-stage pipeline, first learning a reward or preference model from labeled preferences and then performing offline RL on unlabeled data. We revisit offline PbRL through the lens of reward-free representation learning (RFRL) from the zero-shot RL literature, and propose a new training framework that first learns latent successor-measure representations from reward-free offline data, followed by contrastive search and fine-tuning using preference data. Through extensive experiments and ablations, we show that our method achieves superior preference efficiency over offline PbRL baselines. This work is the first to connect RFRL with PbRL, highlighting its potential as a feedback-efficient solution. Our code is publicly available at https://github.com/rl-bandits-lab/FB-PbRL.

Related papers

• Best Policy Learning from Trajectory Preference Feedback [11.896067099790962]
  Preference-based Reinforcement Learning (PbRL) offers a more robust alternative.<n>We study the best policy identification problem in PbRL, motivated by post-training optimization of generative models.<n>We propose Posterior Sampling for Preference Learning ($mathsfPSPL$), a novel algorithm inspired by Top-Two Thompson Sampling.
  arXiv  Detail & Related papers  (2025-01-31T03:55:10Z)
• Unsupervised-to-Online Reinforcement Learning [59.910638327123394]
  Unsupervised-to-online RL (U2O RL) replaces domain-specific supervised offline RL with unsupervised offline RL. U2O RL not only enables reusing a single pre-trained model for multiple downstream tasks, but also learns better representations. We empirically demonstrate that U2O RL achieves strong performance that matches or even outperforms previous offline-to-online RL approaches.
  arXiv  Detail & Related papers  (2024-08-27T05:23:45Z)
• Listwise Reward Estimation for Offline Preference-based Reinforcement Learning [20.151932308777553]
  Listwise Reward Estimation (LiRE) is a novel approach for offline Preference-based Reinforcement Learning (PbRL) LiRE builds on existing PbRL methods by constructing a Ranked List of Trajectories (RLT) Our experiments demonstrate the superiority of LiRE, even with modest feedback budgets and enjoying robustness with respect to the number of feedbacks and feedback noise.
  arXiv  Detail & Related papers  (2024-08-08T03:18:42Z)
• Preference Elicitation for Offline Reinforcement Learning [59.136381500967744]
  We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm.<n>Our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy.
  arXiv  Detail & Related papers  (2024-06-26T15:59:13Z)
• Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning [5.480108613013526]
  We propose a general framework to bridge the gap between reward-based offline RL and preference-based offline RL. Our key insight is transforming preference feedback to scalar rewards via binary reward labeling (BRL) We empirically test our framework on preference datasets based on the standard D4RL benchmark.
  arXiv  Detail & Related papers  (2024-06-14T23:40:42Z)
• RIME: Robust Preference-based Reinforcement Learning with Noisy Preferences [23.414135977983953]
  Preference-based Reinforcement Learning (PbRL) circumvents the need for reward engineering by harnessing human preferences as the reward signal. We present RIME, a robust PbRL algorithm for effective reward learning from noisy preferences.
  arXiv  Detail & Related papers  (2024-02-27T07:03:25Z)
• More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning [58.626683114119906]
  We show that Distributional Reinforcement Learning (DistRL) can obtain second-order bounds in both online and offline RL. Our results are the first second-order bounds for low-rank MDPs and for offline RL.
  arXiv  Detail & Related papers  (2024-02-11T13:25:53Z)
• Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning [66.43003402281659]
  A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset. We design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL. The proposed algorithm does not require any reward information during data collection.
  arXiv  Detail & Related papers  (2023-05-17T15:17:23Z)
• Benchmarks and Algorithms for Offline Preference-Based Reward Learning [41.676208473752425]
  We propose an approach that uses an offline dataset to craft preference queries via pool-based active learning. Our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps.
  arXiv  Detail & Related papers  (2023-01-03T23:52:16Z)
• B-Pref: Benchmarking Preference-Based Reinforcement Learning [84.41494283081326]
  We introduce B-Pref, a benchmark specially designed for preference-based RL. A key challenge with such a benchmark is providing the ability to evaluate candidate algorithms quickly. B-Pref alleviates this by simulating teachers with a wide array of irrationalities.
  arXiv  Detail & Related papers  (2021-11-04T17:32:06Z)
• Preference-based Reinforcement Learning with Finite-Time Guarantees [76.88632321436472]
  Preference-based Reinforcement Learning (PbRL) replaces reward values in traditional reinforcement learning to better elicit human opinion on the target objective. Despite promising results in applications, the theoretical understanding of PbRL is still in its infancy. We present the first finite-time analysis for general PbRL problems.
  arXiv  Detail & Related papers  (2020-06-16T03:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.