Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation

• URL: http://arxiv.org/abs/2608.11980v2
• Date: Sat, 15 Aug 2026 11:55:30 GMT
• Title: Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation
• Abstract summary: Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic tokens and predict the next item by autoregressively generating this token sequence.<n>This paradigm enables a unified generation interface for item IDs, histories, and item text, but it also creates a structured optimization bottleneck during reward-based post-training.<n>We propose HCGRec, a semantic-ID generative recommendation framework that recovers learning signal for such hard training instances.<n>HCGRec substantially improves over supervised fine-tuning and vanilla reward-based post-training, while reducing zero-advantage training samples from over 70% to below 20%.
• Score: 39.5807784132569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic tokens and predict the next item by autoregressively generating this token sequence. This paradigm enables a unified generation interface for item IDs, histories, and item text, but it also creates a structured optimization bottleneck during reward-based post-training: when an early semantic token enters the wrong branch of the item-token space, finite rollout groups rarely reach the ground-truth item, so group-relative optimization receives identical zero rewards and produces no useful advantage. We propose Hint-Conditioned Generative Recommendation (HCGRec), a semantic-ID generative recommendation framework that recovers learning signal for such hard training instances. HCGRec diagnoses each instance with checkpoint rollouts and supplies a minimal target-prefix hint only when the current generator cannot reach the correct item. The model then generates the unhinted suffix under the hinted semantic branch, turning zero-reward groups into informative comparisons over item-token completions. Hinting also changes token identity: hinted prefix tokens are oracle-provided item context, while unhinted suffix tokens are sampled generation actions. We therefore introduce hint-aware credit decomposition, using supervised learning to preserve item-semantic and prefix-structure alignment for hinted tokens and GRPO to optimize the sampled suffix. Experiments on sequential recommendation benchmarks show that HCGRec substantially improves over supervised fine-tuning and vanilla reward-based post-training, while reducing zero-advantage training samples from over 70% to below 20%. The code is accessible at https://github.com/WncFht/GRec.

Related papers

• Preserving Item Semantics for Free: Rethinking Token Initialization in LLM-Based Generative Recommendation [33.094341397930776]
  In generative recommendation systems, items are often represented through semantic IDs (SIDs)<n>Standard vocabulary expansion typically initializes these tokens as random Gaussian vectors, discarding the SIDs' underlying continuous geometry.<n>We propose a simple, parameter-free intervention that initializes SID token embeddings directly from their corresponding centroids in the semantic embedding space.
  arXiv  Detail & Related papers  (2026-08-07T23:33:08Z)
• VaLiDRec: Variable-Length LLM-Aligned Semantic IDs for Generative Recommendation [39.59701442467294]
  VaLiDRec is a generative recommendation framework based on variable-length, LLM-aligned semantic identifiers.<n>To model user preferences, VaLiDRec incorporates graph-aware soft prompts and reformulates recommendation as token-set prediction.<n>Experiments on four real-world datasets show that VaLiDRec consistently outperforms strong and sequential generative recommendation baselines.
  arXiv  Detail & Related papers  (2026-07-28T02:29:10Z)
• SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation [43.98709365191658]
  Generative recommendation treats next-item prediction as autoregressive item-identifier generation.<n>Recent work augments this paradigm with reasoning traces and optimize them via reinforcement learning with verifiable rewards.<n>In large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct.
  arXiv  Detail & Related papers  (2026-05-17T20:53:08Z)
• RAGR: Review-Augmented Generative Recommendation [61.29879991713178]
  Review-Augmented Generative Recommendation (RAGR) is a novel GR framework that incorporates review feedback directly into the generative user sequence.<n>RAGR yields consistent and significant gains over strong GR backbones across all metrics.
  arXiv  Detail & Related papers  (2026-05-17T05:21:23Z)
• CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation [55.783414010717074]
  CoFiRec is a novel generative recommendation framework that decomposes item information into semantic levels.<n>We show that CoFiRec outperforms existing methods, offering a new perspective for generative recommendation.
  arXiv  Detail & Related papers  (2025-11-27T18:59:35Z)
• Pre-training Generative Recommender with Multi-Identifier Item Tokenization [78.87007819266957]
  We propose MTGRec to augment token sequence data for Generative Recommender pre-training.<n>Our approach involves two key innovations: multi-identifier item tokenization and curriculum recommender pre-training.<n>Extensive experiments on three public benchmark datasets demonstrate that MTGRec significantly outperforms both traditional and generative recommendation baselines.
  arXiv  Detail & Related papers  (2025-04-06T08:03:03Z)
• Order-agnostic Identifier for Large Language Model-based Generative Recommendation [94.37662915542603]
  Items are assigned identifiers for Large Language Models (LLMs) to encode user history and generate the next item.<n>Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings.<n>We propose SETRec, which leverages semantic tokenizers to obtain order-agnostic multi-dimensional tokens.
  arXiv  Detail & Related papers  (2025-02-15T15:25:38Z)
• Enhancing Item Tokenization for Generative Recommendation through Self-Improvement [67.94240423434944]
  Generative recommendation systems are driven by large language models (LLMs)<n>Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens.<n>We propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process.
  arXiv  Detail & Related papers  (2024-12-22T21:56:15Z)
• Learning Multi-Aspect Item Palette: A Semantic Tokenization Framework for Generative Recommendation [55.99632509895994]
  We introduce LAMIA, a novel approach for multi-aspect semantic tokenization.<n>Unlike RQ-VAE, which uses a single embedding, LAMIA learns an item palette''--a collection of independent and semantically parallel embeddings.<n>Our results demonstrate significant improvements in recommendation accuracy over existing methods.
  arXiv  Detail & Related papers  (2024-09-11T13:49:48Z)
• Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation [59.37587762543934]
  This paper studies the problem of weakly open-vocabulary semantic segmentation (WOVSS) Existing methods suffer from a granularity inconsistency regarding the usage of group tokens. We propose the prototypical guidance network (PGSeg) that incorporates multi-modal regularization.
  arXiv  Detail & Related papers  (2023-10-29T13:18:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.