Related papers: Tail-Likelihood Reinforcement Learning

Tail-Likelihood Reinforcement Learning

URL: http://arxiv.org/abs/2609.02987v2
Date: Wed, 09 Sep 2026 16:07:49 GMT
Title: Tail-Likelihood Reinforcement Learning
Abstract summary: Tail-Likelihood Reinforcement Learning (TailRL) maximizes the log-probability of exceeding a randomly chosen reward threshold.<n>TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines.
Score: 63.90013981311359
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of-k gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.

Related papers

MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
arXiv Detail & Related papers (2026-05-07T16:20:42Z)
Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training [39.36546278921025]
Reinforcement fine-tuning (RFT) often suffers from emphreward over-optimization, where a policy model hacks the reward signals to achieve high scores.<n>Our theoretical analysis shows that the key lies in reward misspecification at the high-reward tail.<n>While off-policy exemplars are easier to obtain, naively training on them yields a misspecified reward for the policy we aim to align.
arXiv Detail & Related papers (2025-09-25T19:57:39Z)
Residual Reward Models for Preference-based Reinforcement Learning [11.797520525358564]
Preference-based Reinforcement Learning (PbRL) provides a way to learn high-performance policies in environments where the reward signal is hard to specify.<n>PbRL can suffer from slow convergence speed since it requires training in a reward model.<n>We propose a method to effectively leverage prior knowledge with a Residual Reward Model (RRM)
arXiv Detail & Related papers (2025-07-01T09:43:57Z)
Reward Shaping to Mitigate Reward Hacking in RLHF [47.71454266800376]
Preference As Reward (PAR) is a novel approach that leverages the latent preferences embedded within the reward model as the signal for reinforcement learning.<n>On the AlpacaEval 2.0 benchmark, PAR achieves a win rate of at least 5 percentage points higher than competing approaches.
arXiv Detail & Related papers (2025-02-26T02:57:59Z)
RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution [50.171320156632866]
Reinforcement learning from human feedback offers a promising approach to aligning large language models with human preferences.<n>Current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence.<n>We propose a more fine-grained, token-level guidance approach for RL training.
arXiv Detail & Related papers (2024-11-13T02:45:21Z)
The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret [64.04721528586747]
We show that a sufficiently low expected test error of the reward model guarantees low worst-case regret.<n>We then show that similar problems persist even when using policy regularization techniques.
arXiv Detail & Related papers (2024-06-22T06:43:51Z)
RewardsOfSum: Exploring Reinforcement Learning Rewards for Summarisation [7.0471949371778795]
We propose two reward functions for the task of abstractive summarisation. The first function, referred to as RwB-Hinge, dynamically selects the samples for the gradient update. The second function, nicknamed RISK, leverages a small pool of strong candidates to inform the reward.
arXiv Detail & Related papers (2021-06-08T03:30:50Z)
Model-free Policy Learning with Reward Gradients [9.847875182113137]
We develop the textitReward Policy Gradient estimator, a novel approach that integrates reward gradients without learning a model. Our method also boosts the performance of Proximal Policy Optimization on different MuJoCo control tasks.
arXiv Detail & Related papers (2021-03-09T00:14:13Z)
Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping [71.214923471669]
Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL) In this paper, we consider the problem of adaptively utilizing a given shaping reward function. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards.
arXiv Detail & Related papers (2020-11-05T05:34:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.