SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation

• URL: http://arxiv.org/abs/2605.17648v1
• Date: Sun, 17 May 2026 20:53:08 GMT
• Title: SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation
• Abstract summary: Generative recommendation treats next-item prediction as autoregressive item-identifier generation.<n>Recent work augments this paradigm with reasoning traces and optimize them via reinforcement learning with verifiable rewards.<n>In large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct.
• Score: 43.98709365191658
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative recommendation treats next-item prediction as autoregressive item-identifier generation. Specifically, items are encoded as semantic identifiers (SIDs), which are short coarse-to-fine token sequences whose early tokens capture broad semantics and later tokens refine them. Recent work augments this paradigm with reasoning traces and optimizes them via reinforcement learning with verifiable rewards, typically outcome-reward algorithm with exact-match feedback on the generated SID. However, in large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct; when a generated SID mismatches, outcome-reward cannot identify which SID-token prediction caused the mismatch and may penalize matched SID-token positions together with the mismatched position. We identify that the natural unit of credit assignment in this setting is a single reasoning step (one thinking block paired with one SID token). We instantiate this idea in SAPO (Step-Aligned Policy Optimization): rather than broadcasting one advantage to the whole response, SAPO computes a separate group-relative advantage for each reasoning step and applies it only to the corresponding thinking block and SID token. Across three real-world recommendation datasets, SAPO stabilizes reinforcement-learning training and consistently improves over existing generative recommendation baselines, with the largest gains where sparse exact-match feedback makes reasoning-step credit assignment important. Our results suggest that reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output.

Related papers

• EPIC: Explicit Posterior Item Conditioning for Semantic ID Diffusion Recommendation [18.230837696789226]
  Explicit Posterior Item Conditioning introduces explicit item-level competition into SID denoising.<n>EPIC constructs a personalized posterior over feasible candidate items using the current generation context.<n>Experiments on four Amazon benchmarks show consistent improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-09-03T08:21:50Z)
• Learning from Unreachable Rewards: Hint-Conditioned Reinforcement Learning for Generative Recommendation [39.5807784132569]
  Semantic-ID generative recommenders represent each item as a short sequence of discrete semantic tokens and predict the next item by autoregressively generating this token sequence.<n>This paradigm enables a unified generation interface for item IDs, histories, and item text, but it also creates a structured optimization bottleneck during reward-based post-training.<n>We propose HCGRec, a semantic-ID generative recommendation framework that recovers learning signal for such hard training instances.<n>HCGRec substantially improves over supervised fine-tuning and vanilla reward-based post-training, while reducing zero-advantage training samples from over 70% to below 20%.
  arXiv  Detail & Related papers  (2026-08-12T12:13:08Z)
• Hierarchical Residual Policy Optimization for Generative Recommendations [46.030303420508254]
  We propose a post-training framework that converts item-level outcomes into dense, token-aligned learning signals.<n>Experiments on a public dataset and an online A/B test in a large-scale commercial system show consistent gains in session-level utility and key business metrics.
  arXiv  Detail & Related papers  (2026-08-01T16:33:15Z)
• CogRec: Structure-Cognitive Fast-and-Slow Reasoning for Generative Recommendation [29.663400191042996]
  CogRec is a structure-cognitive fast-and-slow reasoning framework.<n>CogRec augments the vertical SID hierarchy with intra-layer semantic graphs and item-level neighborhoods.<n> SID Routing represents recommendation reasoning through layer-wise Match, LateralJump, and Explore operations.
  arXiv  Detail & Related papers  (2026-07-27T13:19:55Z)
• RAGR: Review-Augmented Generative Recommendation [61.29879991713178]
  Review-Augmented Generative Recommendation (RAGR) is a novel GR framework that incorporates review feedback directly into the generative user sequence.<n>RAGR yields consistent and significant gains over strong GR backbones across all metrics.
  arXiv  Detail & Related papers  (2026-05-17T05:21:23Z)
• UniRec: Bridging the Expressive Gap between Generative and Discriminative Recommendation via Chain-of-Attribute [12.89120699793625]
  Generative Recommendation (GR) reframes retrieval and ranking as autoregressive decoding over Semantic IDs (SIDs)<n>We formalize this via Bayes' theorem: ranking by p(y|f,u) is equivalent to ranking by p(f|y,u), which factorizes autoregressively over item features.<n>We propose UniRec with Chain-of-Attribute (CoA) as its core mechanism. CoA prefixes each SID sequence with structured attribute tokens:category, seller, brand, before decoding the SID, recovering the item-side feature crossing that discriminative models exploit.
  arXiv  Detail & Related papers  (2026-04-14T03:13:50Z)
• Reasoning over Semantic IDs Enhances Generative Recommendation [56.83043583765322]
  We propose a two-stage framework that elicits reasoning over SIDs by strengthening SID--language alignment.<n> SIDReasoner further improves recommendation reasoning through outcome-driven reinforced optimization.<n>Our experiments on three real-world datasets demonstrate the effectiveness of our reasoning-augmented SID-based generative recommendation.
  arXiv  Detail & Related papers  (2026-03-24T13:31:48Z)
• End-to-End Semantic ID Generation for Generative Advertisement Recommendation [33.453121305193434]
  We propose a Unified SID generation framework for generative advertisement recommendation.<n>Specifically, we jointly optimize embeddings and SIDs in an end-to-end manner from raw advertising data.<n>Experiments demonstrate that UniSID consistently outperforms state-of-the-art SID generation methods.
  arXiv  Detail & Related papers  (2026-02-11T02:38:26Z)
• Rethinking Generative Recommender Tokenizer: Recsys-Native Encoding and Semantic Quantization Beyond LLMs [17.944727019161878]
  ReSID is a principled, SID framework that recommend learning from the perspective of information preservation and sequential predictability.<n>It consistently outperforms strong sequential and SID-based generative baselines by an average of over 10%, while reducing tokenization cost by up to 122x.
  arXiv  Detail & Related papers  (2026-02-02T17:00:04Z)
• Differentiable Semantic ID for Generative Recommendation [65.83703273297492]
  Generative recommendation provides a novel paradigm in which each item is represented by a discrete semantic ID (SID) learned from rich content.<n>In practice, SIDs are typically optimized only for content reconstruction rather than recommendation accuracy.<n>A natural approach is to make semantic indexing differentiable so that recommendation gradients can directly influence SID learning.<n>We propose DIGER, a first step toward effective differentiable semantic IDs for generative recommendation.
  arXiv  Detail & Related papers  (2026-01-27T15:34:11Z)
• DiffGRM: Diffusion-based Generative Recommendation Model [63.35379395455103]
  Generative recommendation (GR) is an emerging paradigm that represents each item via a tokenizer as an n-digit semantic ID (SID)<n>We propose DiffGRM, a diffusion-based GR model that replaces the autoregressive decoder with a masked discrete diffusion model (MDM)<n> Experiments show consistent gains over strong generative and discriminative recommendation baselines on multiple datasets.
  arXiv  Detail & Related papers  (2025-10-21T03:23:32Z)
• Neighbour Consistency Guided Pseudo-Label Refinement for Unsupervised Person Re-Identification [80.98291772215154]
  Unsupervised person re-identification (ReID) aims at learning discriminative identity features for person retrieval without any annotations. Recent advances accomplish this task by leveraging clustering-based pseudo labels. We propose a Neighbour Consistency guided Pseudo Label Refinement framework.
  arXiv  Detail & Related papers  (2022-11-30T09:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.