Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner

• URL: http://arxiv.org/abs/2609.12651v1
• Date: Fri, 11 Sep 2026 09:55:06 GMT
• Title: Distortion of AI Alignment Revisited: RLHF is a Decent Utilitarian Aligner
• Abstract summary: We present a fine-grained analysis of the distortion of RLHF with reward clipping.<n>We show that when there is no distribution mismatch, RLHF achieves the optimal distortion of $O()$ up to a constant.
• Score: 25.64947652322614
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While Reinforcement Learning from Human Feedback (RLHF) is the standard paradigm for aligning large language models with human preferences, its effectiveness in pluralistic settings has been called into question. Notably, recent work by Gölz et al. (2025) demonstrated that the \textit{distortion} -- defined as the multiplicative gap between the average user utility of the RLHF policy and the optimal average utility -- can scale exponentially with the Bradley-Terry temperature parameter $β$ when users have heterogeneous preferences. In this work, we present a fine-grained analysis of the distortion of RLHF with reward clipping and demonstrate that such exponential degradation is not a fundamental property of the algorithm but rather a consequence of distribution mismatch between the distribution generating preference data ($μ$) and the KL reference policy ($π_{\mathrm{ref}}$). To this end, we establish tight upper and lower bounds on the distortion of RLHF across multiple regimes of the KL regularization strength. We show that in a representative regime, under the Bradley-Terry model, the distortion is $\tildeΘ(βB + β)$, where $B$ is an upper bound on the log density ratio between $μ$ and $π_{\mathrm{ref}}$. In particular, when there is no distribution mismatch (i.e., $μ= π_{\mathrm{ref}}$), RLHF achieves the optimal distortion of $O(β)$ up to a constant. Our results suggest that, to reasonably maximize average utility with RLHF, it is preferable to use on-policy sampled preference data or to fine-tune before RLHF on data from a source close to $μ$.

Related papers

• Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret [71.69884486156359]
  We study episodic RL from emphmulti-source imperfect preferences through a cumulative imperfection budget.<n>We propose a unified algorithm with regret $tildeO(sqrtK/M+)$, which exhibits a best-of-both-regimes behavior.
  arXiv  Detail & Related papers  (2026-03-20T19:34:53Z)
• Regularized Online RLHF with Generalized Bilinear Preferences [68.44113000390544]
  We consider the problem of contextual online RLHF with general preferences.<n>We adopt the Generalized Bilinear Preference Model to capture preferences via low-rank, skew-symmetric matrices.<n>We prove that the dual gap of the greedy policy is bounded by the square of the estimation error.
  arXiv  Detail & Related papers  (2026-02-26T15:27:53Z)
• Sampling non-log-concave densities via Hessian-free high-resolution dynamics [8.014749318189851]
  We study the problem of sampling from a target distribution $(q)prop e-U(q) of the Hessian-free-resolution (HFHR) process.<n>We show that, under an additional assumption that $blana U$ linear growth has at infinity, the contraction for HFHR dynamics is strictly better than that KLD is.
  arXiv  Detail & Related papers  (2026-01-06T05:38:09Z)
• Offline and Online KL-Regularized RLHF under Differential Privacy [15.993352181567872]
  We study the offline and online settings of reinforcement learning from human feedback.<n>In the offline setting, we design an algorithm based on the principle of pessimism.<n>In the online setting, we are the first to theoretically investigate the problem of KL-regularized RLHF with LDP.
  arXiv  Detail & Related papers  (2025-10-15T13:04:19Z)
• DRO-REBEL: Distributionally Robust Relative-Reward Regression for Fast and Efficient LLM Alignment [0.0]
  Reinforcement learning with human feedback (RLHF) has become crucial for aligning Large Language Models (LLMs) with human intent.<n>We introduce DRO-REBEL, a unified family of robust REBEL updates with type-$p$ Wasserstein, KL, and $chi2$ ambiguity sets.<n>Using Fenchel duality, each update reduces to a simple relative-reward regression, preserving scalability and avoiding PPO-style clipping or auxiliary value networks.
  arXiv  Detail & Related papers  (2025-09-23T14:49:48Z)
• Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences? [20.004349891563706]
  After pre-training, large language models are aligned with human preferences based on pairwise comparisons.<n>We introduce an alignment method's distortion: the worst-case ratio between the optimal achievable average utility, and the average utility of the learned policy.
  arXiv  Detail & Related papers  (2025-05-29T17:59:20Z)
• Robust Reinforcement Learning from Corrupted Human Feedback [86.17030012828003]
  Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. We propose a robust RLHF approach -- $R3M$, which models the potentially corrupted preference label as sparse outliers. Our experiments on robotic control and natural language generation with large language models (LLMs) show that $R3M$ improves robustness of the reward against several types of perturbations to the preference data.
  arXiv  Detail & Related papers  (2024-06-21T18:06:30Z)
• Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward [66.81579829897392]
  We propose a novel offline reinforcement learning algorithm called Pessimistic vAlue iteRaTion with rEward Decomposition (PARTED) PARTED decomposes the trajectory return into per-step proxy rewards via least-squares-based reward redistribution, and then performs pessimistic value based on the learned proxy reward. To the best of our knowledge, PARTED is the first offline RL algorithm that is provably efficient in general MDP with trajectory-wise reward.
  arXiv  Detail & Related papers  (2022-06-13T19:11:22Z)
• Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximation [107.54516740713969]
  We study human-in-the-loop reinforcement learning (RL) with trajectory preferences. Instead of receiving a numeric reward at each step, the agent only receives preferences over trajectory pairs from a human overseer. We propose the first optimistic model-based algorithm for PbRL with general function approximation.
  arXiv  Detail & Related papers  (2022-05-23T09:03:24Z)
• An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gap [66.75488143823337]
  We show that an exponential sample complexity lower bound still holds even if a constant suboptimality gap is assumed. Perhaps surprisingly, this implies an exponential separation between the online RL setting and the generative model setting.
  arXiv  Detail & Related papers  (2021-03-23T17:05:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.