The Representation-Rationalizability Tradeoff in Reward Learning

• URL: http://arxiv.org/abs/2606.00291v1
• Date: Fri, 29 May 2026 19:22:41 GMT
• Title: The Representation-Rationalizability Tradeoff in Reward Learning
• Abstract summary: We train RLHF as a social-choice problem, but assume a pre-enumerated finite set of candidate responses for each prompt.<n>Modern pipelines score responses through a learned representation $(x,y)$ before a scalar head, so $$ determines which responses are treated as distinguishable alternatives.<n>We show that the excess cross-entropy loss of any reward built on $$ decomposes exactly into a representational term, which a richer $$ shrinks, and an aggregation term, which a richer $$ enlarges.
• Score: 24.1466886968784
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In RLHF, each training example contains a prompt $x$ and two candidate responses $y,y'$, and annotators provide pairwise preferences between these responses. The learning problem is to convert these heterogeneous pairwise judgments into a single scalar reward $r(x,y)$ that measures response quality for each prompt. Classical social choice implies an impossibility because heterogeneous annotator samples can induce pooled preferences with Condorcet cycles, so no scalar reward can evaluate all compared response pairs consistently. A growing literature analyzes RLHF as a social-choice problem, but usually assumes a fixed finite set of alternatives, i.e., a pre-enumerated finite set of candidate responses for each prompt. Modern pipelines instead score responses through a learned representation $φ(x,y)$ before a scalar head, so $φ$ determines which responses are treated as distinguishable alternatives and which comparisons are visible to the reward model. Once this embedding is part of the problem, the impossibility results from social choice theory become a tradeoff. We show that the excess cross-entropy loss of any reward built on $φ$ decomposes exactly into a representational term, which a richer $φ$ shrinks, and an aggregation term, which a richer $φ$ enlarges by exposing more comparisons that no scalar can rank consistently. The same results extend to direct preference optimization (DPO), and jointly training the embedding and the reward cannot guarantee to recover the sweet spot of this tradeoff. Experiments on synthetic data and real preference datasets corroborate our results.

Related papers

• Relational Response Fields: A General Theory of Black-Box LLM Response Consistency and Recovery [0.0]
  Black-box language-model reliability is commonly pursued by sampling, prompting, voting, verifying, or iteratively revising individual answers.<n>We represent responses to typed transformations of a query as a emphrelational response field (RRF)<n>anchors encode how valid responses must change under paraphrase, scaling, decomposition, or other task symmetries.<n>For relation operator $D$, anchor operator $A$, and at most $k$ corrupted response nodes, we identify $_k(D,A)$ as the intrinsic difficulty of black-box response recovery.
  arXiv  Detail & Related papers  (2026-08-05T07:43:41Z)
• The Sample Complexity of Multiclass and Sparse Contextual Bandits [106.74652380822778]
  We aim to identify an approximately optimal policy from a given class based on bandit feedback.<n>Motivated by bandit multiclass classification with zero-one rewards, we focus on the emph$s$-sparse setting.<n>We show that, with $s$-sparse rewards, the induced model class admits a sharp DEC bound that scales with $s$ and directly yields the optimal rate.
  arXiv  Detail & Related papers  (2026-05-28T09:12:20Z)
• Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret [71.69884486156359]
  We study episodic RL from emphmulti-source imperfect preferences through a cumulative imperfection budget.<n>We propose a unified algorithm with regret $tildeO(sqrtK/M+)$, which exhibits a best-of-both-regimes behavior.
  arXiv  Detail & Related papers  (2026-03-20T19:34:53Z)
• Likelihood-Based Reward Designs for General LLM Reasoning [17.640897774014707]
  Fine-tuning large language models via reinforcement learning requires a specific reward function, often binary, for each benchmark.<n>Here, we investigate rewards derived from the probability or log-probability of emitting the reference answer.<n>We find that using the log-probability of the reference answer as the reward for chain-of-thought learning is the only option that performs well in all setups.
  arXiv  Detail & Related papers  (2026-02-03T20:04:21Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Reasoning without Regret [4.07926531936425]
  We introduce emphBackwards Adaptive Reward Shaping (BARS), a no-regret framework that converts sparse outcomes-based rewards into effective procedure-based signals.<n>Our analysis, based on generic chaining, continuous scaling limits, and non-linear Feynman-Kac bounds, connects recent outcome-based methods' empirical successes with the benefits of intermediate supervision.
  arXiv  Detail & Related papers  (2025-04-14T00:34:20Z)
• Information-Theoretic Reward Decomposition for Generalizable RLHF [51.550547285296794]
  We decompose the reward value into two independent components: prompt-free reward and prompt-related reward.<n>We propose a new reward learning algorithm by prioritizing data samples based on their prompt-free reward values.
  arXiv  Detail & Related papers  (2025-04-08T13:26:07Z)
• Robust Reinforcement Learning from Corrupted Human Feedback [86.17030012828003]
  Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. We propose a robust RLHF approach -- $R3M$, which models the potentially corrupted preference label as sparse outliers. Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R3M$ improves robustness of the reward against several types of perturbations to the preference data.
  arXiv  Detail & Related papers  (2024-06-21T18:06:30Z)
• Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis [16.288866201806382]
  We develop a model-free RLHF best policy identification algorithm, called $mathsfBSAD$, without explicit reward model inference.<n>The algorithm identifies the optimal policy directly from human preference information in a backward manner.
  arXiv  Detail & Related papers  (2024-06-11T17:01:41Z)
• Towards Theoretical Understanding of Inverse Reinforcement Learning [45.3190496371625]
  Inverse reinforcement learning (IRL) denotes a powerful family of algorithms for recovering a reward function justifying the behavior demonstrated by an expert agent. In this paper, we make a step towards closing the theory gap of IRL in the case of finite-horizon problems with a generative model.
  arXiv  Detail & Related papers  (2023-04-25T16:21:10Z)
• Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward [66.81579829897392]
  We propose a novel offline reinforcement learning algorithm called Pessimistic vAlue iteRaTion with rEward Decomposition (PARTED) PARTED decomposes the trajectory return into per-step proxy rewards via least-squares-based reward redistribution, and then performs pessimistic value based on the learned proxy reward. To the best of our knowledge, PARTED is the first offline RL algorithm that is provably efficient in general MDP with trajectory-wise reward.
  arXiv  Detail & Related papers  (2022-06-13T19:11:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.