Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation

• URL: http://arxiv.org/abs/2606.08480v1
• Date: Sun, 07 Jun 2026 06:51:18 GMT
• Title: Adaptive Loss Balancing for Noise-Robust GRPO in Generative Recommendation
• Abstract summary: Reinforcement learning (RL) presents a promising avenue for enhancing generative recommendation beyond supervised imitation.<n>In practice, production rankers, the widely adopted reward models, are trained on exposure-biased logs, leading to sample-dependent inaccuracies.<n>We introduce AdaGRPO, a novel framework that treats reward-guided optimization as selective admission rather than uniform pressure.
• Score: 9.508867853240469
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) presents a promising avenue for enhancing generative recommendation beyond supervised imitation, leveraging reward signals to guide policy improvement. However, its efficacy is critically contingent on the trustworthiness of the reward model for the samples it evaluates. In practice, production rankers, the widely adopted reward models, are trained on exposure-biased logs, leading to sample-dependent inaccuracies that violate this assumption. Our stratified analysis uncovers a consistent pattern: reward guidance is most beneficial when the policy exhibits uncertainty and the ranker can effectively discriminate the ground-truth item from rollout negatives. On other samples, the reward signal is either negligible or detrimental, highlighting the risk of uniform RL application. To address such an issue, we introduce AdaGRPO, a novel framework that treats reward-guided optimization as selective admission rather than uniform pressure. Training is anchored in supervised negative log-likelihood, while the GRPO objective is gated by a binary, per-sample clip determined by two rollout diagnostics: policy-side difficulty and reward discriminability. Instances failing either diagnostic default to pure supervision, ensuring stability and mitigating the amplification of noisy gradients. We validate AdaGRPO on a large-scale e-commerce dataset. At the best intermediate checkpoint, it elevates HR@10 from 11.01% to 12.18% while constraining hallucination below 0.22%, and maintains robustness at the final checkpoint (HR@10 11.63%, hallucination 0.27%), outperforming fixed NLL--GRPO mixtures across the retrieval--validity frontier. In production A/B tests, AdaGRPO achieves statistically significant gains in click-through rate and dwell time, confirming its practical utility.

Related papers

• When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming [0.0]
  We present an empirical failure-mode study of a compact RLHF pipeline with proximal policy optimization (PPO)<n>We classify matched transitions between checkpoints using the directions of the learned reward, judge scores, and average judge score.<n>A pre-transition logistic model predicts future row-level reward hacking with ROC-AUC 0.821, and row-level analysis finds localized reward hacking that checkpoint averages miss in 3 of 12 settings.
  arXiv  Detail & Related papers  (2026-06-02T06:55:52Z)
• VeriGate: Verifier-Gated Step-Level Supervision for GRPO [51.26100506256885]
  Group Relative Policy Optimization is an effective recipe for training reasoning models with verifier-based outcome rewards.<n>We present VeriGate, a verifier-gated extension of GRPO that addresses these limitations with three design choices.<n>We show that VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively.
  arXiv  Detail & Related papers  (2026-05-28T18:20:32Z)
• Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation [13.272542054938258]
  We introduce the Advantage Collapse Rate (ACR), the first metric quantifying the proportion of training batches with ineffective gradients.<n>We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples.<n>AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales.
  arXiv  Detail & Related papers  (2026-05-20T12:57:37Z)
• ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization [14.900223489465683]
  Large Language Models (LLMs) utilize Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains.<n>These domains often rely on based auto-raters to provide granular, multi-tier discrete rewards.<n>We introduce $textbfO$ptimization ($textbfOOO$), a framework that structurally isolates evaluation noise by decomposing discrete rewards into a sequence of ordinal binary indicators.
  arXiv  Detail & Related papers  (2026-05-12T19:17:14Z)
• RVPO: Risk-Sensitive Alignment via Variance Regularization [13.192921543523283]
  We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation.<n>We show via Taylor expansion that a LogSumExp (SoftMin) operator effectively acts as a smooth variance penalty.<n>By preventing the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench.
  arXiv  Detail & Related papers  (2026-05-07T06:43:05Z)
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training [69.32453275232662]
  We show that a learned critic can inject estimation noise that exceeds the state signal it captures, increasing rather than reducing advantage variance.<n>We propose Explained Variance Policy Optimization (EVPO), which monitors batch-level EV at each training step and adaptively switches between critic-based and batch-mean advantage estimation.
  arXiv  Detail & Related papers  (2026-04-21T14:07:39Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• GIPO: Gaussian Importance Sampling Policy Optimization [12.306486689840774]
  GIPO is proposed as a policy optimization objective based on truncated importance sampling.<n>It replaces hard clipping with a log-ratio-based Gaussian trust weight to damp extreme importance ratios.<n>GIPO achieves state-of-the-art performance among clipping-based baselines across a wide range of replay buffer sizes.
  arXiv  Detail & Related papers  (2026-03-04T11:34:59Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• Repurposing Synthetic Data for Fine-grained Search Agent Supervision [81.95597592711688]
  LLM-based search agents are increasingly trained on entity-centric synthetic data.<n> prevailing training methods discard this rich entity information, relying instead on sparse, outcome-based rewards.<n>We introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function.
  arXiv  Detail & Related papers  (2025-10-28T17:50:40Z)
• NGRPO: Negative-enhanced Group Relative Policy Optimization [8.641009168869195]
  A representative RLVR algorithm, GRPO, suffers from a critical limitation when all responses within a group are either entirely correct or entirely incorrect.<n>This is particularly problematic for homogeneously incorrect groups, where GRPO's advantage function yields a value of zero.<n>We propose NGRPO, an algorithm designed to convert homogeneous errors into robust learning signals.
  arXiv  Detail & Related papers  (2025-09-23T09:38:10Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.