ReCast: Recasting Learning Signals for Reinforcement Learning in Generative Recommendation

• URL: http://arxiv.org/abs/2604.22169v1
• Date: Fri, 24 Apr 2026 02:44:36 GMT
• Title: ReCast: Recasting Learning Signals for Reinforcement Learning in Generative Recommendation
• Abstract summary: We propose ReCast, a repair-then-contrast learning-signal framework.<n>We show that ReCast restores minimal learnability for all-zero groups.<n>We also show that ReCast mitigates the persistent all-zero / single-hit regime.
• Score: 18.825912740441858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generic group-based RL assumes that sampled rollout groups are already usable learning signals. We show that this assumption breaks down in sparse-hit generative recommendation, where many sampled groups never become learnable at all. We propose ReCast, a repair-then-contrast learning-signal framework that first restores minimal learnability for all-zero groups and then replaces full-group reward normalization with a boundary-focused contrastive update on the strongest positive and the hardest negative. ReCast leaves the outer RL framework unchanged, modifies only within-group signal construction, and partially decouples rollout search width from actor-side update width. Across multiple generative recommendation tasks, ReCast consistently outperforms OpenOneRec-RL, achieving up to 36.6% relative improvement in Pass@1. Its matched-budget advantage is substantially larger: ReCast reaches the baseline's target performance with only 4.1% of the rollout budget, and this advantage widens with model scale. The same design also yields direct system-level gains, reducing actor-side update time by 16.60x, lowering peak allocated memory by 16.5%, and improving actor MFU by 14.2%. Mechanism analysis shows that ReCast mitigates the persistent all-zero / single-hit regime, restores learnability when natural positives are scarce, and converts otherwise wasted rollout budget into more stable policy updates. These results suggest that, for generative recommendation, the decisive RL problem is not only how to assign rewards, but how to construct learnable optimization events from sparse, structured supervision.

Related papers

• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation [13.272542054938258]
  We introduce the Advantage Collapse Rate (ACR), the first metric quantifying the proportion of training batches with ineffective gradients.<n>We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples.<n>AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales.
  arXiv  Detail & Related papers  (2026-05-20T12:57:37Z)
• Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR [25.35547462790362]
  Group-based policy optimization methods typically allocate a fixed number of rollouts to every prompt.<n>We introduce hit utility, the posterior probability that at least one rollout in a proposed additional allocation for a prompt will be correct.<n>We propose Hit-Utility Optimal Rollout Allocation (HORA), a learning-free rollout allocation policy.
  arXiv  Detail & Related papers  (2026-05-08T01:42:25Z)
• Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regime [28.346931878148524]
  Prefix Sampling (PS) replays self-generated trajectory prefixes to steer skewed groups toward this regime.<n>PS reaches the baseline high-score regime within evaluation variability while delivering 2.01x and 1.55x end-to-end wall-clock speedups.<n>AIME 2025 experiments on 4B and 8B show the same pass-rate-control pattern.
  arXiv  Detail & Related papers  (2026-05-06T16:44:38Z)
• ResRL: Boosting LLM Reasoning via Negative Sample Projection Residual Reinforcement Learning [60.411097231348975]
  Reinforcement Learning with Verifiable Rewards (RLVR) enhances reasoning of Large Language Models (LLMs)<n>This paper proposes negative sample projection Residual Reinforcement Learning (ResRL) that decouples similar semantic distributions among positive and negative responses.
  arXiv  Detail & Related papers  (2026-05-01T03:57:44Z)
• Off-Policy Value-Based Reinforcement Learning for Large Language Models [25.962820072445222]
  ReVal is a Bellman-update-based method that combines stepwise signals capturing internal consistency with trajectory-level signals derived from outcome verification.<n>On DeepSeek-R1-Distill-1.5B, ReVal improves training efficiency and achieves improvement of 2.7% in AIME24 and 4.5% in out-of-domain benchmark GPQA over GRPO.
  arXiv  Detail & Related papers  (2026-03-24T15:55:02Z)
• REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge [83.2858110368572]
  We propose textbfREAL (underlineREgression-underlineAware Reinforcement underlineLThought), a principled RL framework designed to optimize regression rewards.<n>We show that REAL consistently outperforms both regression-aware SFT baselines and standard RL methods.
  arXiv  Detail & Related papers  (2026-03-17T21:19:08Z)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving [75.29519604607111]
  Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning performance of large language models (LLMs)<n>We show that when the initial direction or quality of the chain-of-thought (CoT) is suboptimal, the model often fails to reach the correct answer.<n>We introduce Reinforcement Learning with Re-solving (Re$2$), in which LLMs learn to flexibly abandon unproductive reasoning paths and restart the solution process when necessary.
  arXiv  Detail & Related papers  (2026-03-07T13:17:46Z)
• Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learning [37.10958874118462]
  Reinforcement learning (RL) plays a central role in large language model (LLM) post-training.<n>In GRPO, each query prompts the LLM to generate a group of rollouts with a fixed group size $N$.<n>When all rollouts in a group share the same outcome, either all correct or all incorrect, the group-normalized advantages become zero.<n>We introduce Adaptive Efficient Rollout Optimization (AERO), an enhancement of GRPO.
  arXiv  Detail & Related papers  (2026-02-15T23:14:05Z)
• From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning [7.6602542594279335]
  We propose Reinforcement Learning with Relative Rewards to shift reward shaping from absolute scoring to relative ranking.<n>We show that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks.
  arXiv  Detail & Related papers  (2026-01-30T15:07:06Z)
• RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization [65.23034604711489]
  We introduce RLoop, a self-improving framework for training large reasoning models.<n>RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset.<n>Our experiments show RLoops forgetting and substantially improves generalization, boosting average accuracy by 9% and pass@32 by over 15% compared to vanilla RL.
  arXiv  Detail & Related papers  (2025-11-06T11:27:16Z)
• Reinforced Preference Optimization for Recommendation [28.87206911186567]
  We propose Reinforced Preference Optimization for Recommendation (ReRe) for generative recommenders.<n>ReRe incorporates constrained beam search to improve sampling efficiency and diversify hard negatives.<n>We show that ReRe consistently outperforms both traditional and LLM-based recommenders in ranking performance.
  arXiv  Detail & Related papers  (2025-10-14T07:04:33Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.