ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization

• URL: http://arxiv.org/abs/2605.12667v2
• Date: Thu, 14 May 2026 18:04:15 GMT
• Title: ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization
• Abstract summary: Large Language Models (LLMs) utilize Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains.<n>These domains often rely on based auto-raters to provide granular, multi-tier discrete rewards.<n>We introduce $textbfO$ptimization ($textbfOOO$), a framework that structurally isolates evaluation noise by decomposing discrete rewards into a sequence of ordinal binary indicators.
• Score: 14.900223489465683
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The alignment of Large Language Models (LLMs) utilizes Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains such as long-form question answering and open-ended instruction following. These domains often rely on LLM based auto-raters to provide granular, multi-tier discrete rewards (e.g., 1-10 rubrics) that are inherently stochastic due to prompt sensitivity and sampling randomness. We empirically verify the stochasticity of auto-raters that can propagate and corrupt standard advantage estimators like GRPO and MaxRL, as a noisy reward samples can skew normalization statistics and degrade the global learning signal. Empirically, sampling more rewards and taking majority voting may reduce the noise and improve performance, but this approach is computationally expensive. To address this bottleneck, we introduce $\textbf{O}$rdinal $\textbf{D}$ecomposition for $\textbf{R}$obust $\textbf{P}$olicy $\textbf{O}$ptimization ($\textbf{ODRPO}$), a framework that structurally isolates evaluation noise by decomposing discrete rewards into a sequence of ordinal binary indicators. By independently computing and accumulating advantages across these progressively challenging success thresholds, ODRPO prevents outlier evaluations from corrupting the global update while establishing an implicit, variance-aware learning curriculum. Empirically, ODRPO achieves robust performance on Qwen2.5-7B and Qwen3-4B models, outperforming baselines with relative improvements of upto 14.8% on FACTS-grounding-v2 and 7.5% on Alpaca-Evals. Critically, these gains are achieved with negligible training-time overhead, as ODRPO requires no additional compute per step compared to standard estimators. Supported by theoretical analysis confirming its optimization stability, ODRPO provides a scalable and robust framework for aligning models within the noisy, discrete evaluation landscape of modern RLAIF.

Related papers

• Ratio-Variance Regularized Policy Optimization [64.95520246570446]
  We show that explicitly constraining the policy ratio variance provides a principled local approximation to trust-region constraints.<n>We introduce $bf R2bf VPO$ (Ratio-Variance Regularized Policy Optimization), which implements this constraint via a primal-dual optimization framework.
  arXiv  Detail & Related papers  (2026-05-26T09:53:42Z)
• Hölder Policy Optimisation [26.521180498291717]
  textbfHlderPO is a generalised policy optimisation framework.<n>It unifies token-level probability aggregation via the Hlder mean.<n>It achieves a state-of-the-art average accuracy of $54.9%$ across multiple mathematical benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T12:45:03Z)
• Relative Score Policy Optimization for Diffusion Language Models [29.344961499429257]
  Diffusion large language models (dLLMs) offer a promising route to parallel and efficient text generation.<n>The lack of tractable sequence-level log-ratios forces existing methods to rely on high-variance ELBO-based approximations.<n>We propose textbfRelative textbfScore textbfPolicy textbfOptimization (RSPO) to calibrate noisy likelihood estimates in dLLMs.
  arXiv  Detail & Related papers  (2026-05-11T08:58:40Z)
• Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States [14.658333957936769]
  We introduce Policy Optimization with Internal State Value Estimation.<n>It obtains a baseline at negligible cost by using the policy model's internal signals already computed during the policy forward pass.<n>A lightweight probe predicts the expected verifiable reward from the hidden states of the prompt and generated trajectory.
  arXiv  Detail & Related papers  (2026-05-08T10:49:36Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• $V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts [81.48669089692189]
  Generalist Value Models (such as $V_0.5$) achieve pre-trained value estimation by explicitly encoding model capabilities in-context.<n>We propose $V_0.5$, which adaptively fuses the baseline predicted by such value model with the empirical mean derived from sparse rollouts.<n>$V_0.5$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
  arXiv  Detail & Related papers  (2026-03-11T14:57:41Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• Difficulty-Estimated Policy Optimization [38.86673795561421]
  We propose Difficulty-Estimated Policy Optimization (DEPO), a novel framework designed to optimize the efficiency and robustness of reasoning alignment.<n>Our approach significantly lowers the computational barrier for training high-performance reasoning models, offering a more sustainable path for reasoning scaling.
  arXiv  Detail & Related papers  (2026-02-06T04:12:23Z)
• Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning [48.34492357368989]
  We propose a primal-dual framework that supports stable on-policy learning and enables principled off-policy data reuse.<n>$R2VPO$ achieves superior performance with average relative gains of up to 17% over strong clipping-based baselines.
  arXiv  Detail & Related papers  (2026-01-06T14:01:42Z)
• G$^2$RPO: Granular GRPO for Precise Reward in Flow Models [74.21206048155669]
  We propose a novel Granular-GRPO (G$2$RPO) framework that achieves precise and comprehensive reward assessments of sampling directions.<n>We introduce a Multi-Granularity Advantage Integration module that aggregates advantages computed at multiple diffusion scales.<n>Our G$2$RPO significantly outperforms existing flow-based GRPO baselines.
  arXiv  Detail & Related papers  (2025-10-02T12:57:12Z)
• Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance [52.65461207786633]
  Policy-based Reinforcement Learning from Human Feedback is essential for aligning large language models with human preferences.<n>It requires joint training of an actor and critic with a pretrained, fixed reward model for guidance.<n>We propose textbfDecoupled Value Policy Optimization (DVPO), a lean framework that replaces traditional reward modeling with a pretrained emphglobal value model (GVM)
  arXiv  Detail & Related papers  (2025-02-24T08:11:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.