Hierarchical Residual Policy Optimization for Generative Recommendations

• URL: http://arxiv.org/abs/2608.00750v1
• Date: Sat, 01 Aug 2026 16:33:15 GMT
• Title: Hierarchical Residual Policy Optimization for Generative Recommendations
• Abstract summary: We propose a post-training framework that converts item-level outcomes into dense, token-aligned learning signals.<n>Experiments on a public dataset and an online A/B test in a large-scale commercial system show consistent gains in session-level utility and key business metrics.
• Score: 46.030303420508254
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generative recommenders select items by autoregressively decoding semantic identifiers (SIDs), whose token positions induce a coarse-to-fine hierarchy over the item space. In practice, SID decoders are trained via supervised next-token prediction, which imitates logged trajectories rather than directly optimizing downstream utility. This motivates post-training with outcome feedback to guide decoding toward higher utility. However, logged feedback is only observed for the final exposed item, causing most post-training methods to operate at the item level and broadcast the same terminal signal across all SID tokens. As a result, token-level credit assignment becomes sparse, high-variance, and layer-dependent. To this end, we propose Hierarchical Residual Policy Optimization (HRPO), a post-training framework that converts item-level outcomes into dense, token-aligned learning signals for conservative token-wise improvement. Specifically, HRPO first estimates SID prefix-level utilities via group-wise reward smoothing over feature-based user clusters. It then decomposes these utilities into residual token credits and accumulates them into credit-to-go signals. Finally, Residual-Return Policy Optimization (RRPO) optimizes the residual credits using clipped updates, group-normalized advantages, and KL regularization to preserve stability. Experiments on a public dataset and an online A/B test in a large-scale commercial system show consistent gains in session-level utility and key business metrics. Source code and the archived artifact are available for reproduction.

Related papers

• EchoRec: Multi-Item Prediction-Empowered Generative Recommendation via Cycle-Consistent Preference Alignment [57.08595331216049]
  Generative recommendation autoregressively generates the semantic IDs of the target item, unifying preference modeling and index retrieval within the shared token space.<n>Recent attempts have introduced Multi-Token Prediction (MTP), yet they primarily inherit its efficiency merit, leaving its potential as dense supervision unexplored.<n>We propose EchoRec, which empowers MTP with cycle-consistent holistic preference alignment across multi-horizon for generative recommendation.
  arXiv  Detail & Related papers  (2026-08-14T06:59:12Z)
• Guess Where You Go: Generative Next Point-of-Interest Recommendation in Amap [12.259216278301743]
  We propose an end-to-end industrial framework that integrates semantic identifier (SID) generation with generative Point-of-Interest (POI) recommendation.<n>Gwhere is deployed in Amap's homepage service under high-concurrency and low-latency constraints.<n>Long-term online A/B tests show improvements of 5.83% in P-CTR and 6.20% in U-CTR over the production baseline.
  arXiv  Detail & Related papers  (2026-07-13T15:44:06Z)
• SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation [43.98709365191658]
  Generative recommendation treats next-item prediction as autoregressive item-identifier generation.<n>Recent work augments this paradigm with reasoning traces and optimize them via reinforcement learning with verifiable rewards.<n>In large-catalog recommendation, exact-match feedback on the generated SID only reports whether the final item is correct.
  arXiv  Detail & Related papers  (2026-05-17T20:53:08Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching [20.353416189523006]
  We show how to recover token-level optimality using only standard sequence-level pairwise comparisons.<n>We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization.
  arXiv  Detail & Related papers  (2026-05-12T15:44:33Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Signal Reshaping for GRPO in Weak-Feedback Agentic Code Repair [36.31472731207028]
  We study signal reshaping for standard GRPO under such feedback.<n>Experiments show a clear end-to-end gain: full signal-reshaped GRPO improves strict compile-and-semantic accuracy.
  arXiv  Detail & Related papers  (2026-05-08T05:41:25Z)
• UniRec: Bridging the Expressive Gap between Generative and Discriminative Recommendation via Chain-of-Attribute [12.89120699793625]
  Generative Recommendation (GR) reframes retrieval and ranking as autoregressive decoding over Semantic IDs (SIDs)<n>We formalize this via Bayes' theorem: ranking by p(y|f,u) is equivalent to ranking by p(f|y,u), which factorizes autoregressively over item features.<n>We propose UniRec with Chain-of-Attribute (CoA) as its core mechanism. CoA prefixes each SID sequence with structured attribute tokens:category, seller, brand, before decoding the SID, recovering the item-side feature crossing that discriminative models exploit.
  arXiv  Detail & Related papers  (2026-04-14T03:13:50Z)
• Differentiable Semantic ID for Generative Recommendation [65.83703273297492]
  Generative recommendation provides a novel paradigm in which each item is represented by a discrete semantic ID (SID) learned from rich content.<n>In practice, SIDs are typically optimized only for content reconstruction rather than recommendation accuracy.<n>A natural approach is to make semantic indexing differentiable so that recommendation gradients can directly influence SID learning.<n>We propose DIGER, a first step toward effective differentiable semantic IDs for generative recommendation.
  arXiv  Detail & Related papers  (2026-01-27T15:34:11Z)
• RLPO: Residual Listwise Preference Optimization for Long-Context Review Ranking [50.709454968853954]
  Pointwise scoring is efficient but often fails to account for list-level interactions.<n>Listwise approaches can leverage global context, yet they are computationally expensive and become unstable as candidate lists grow.<n>We propose Residual Listwise Preference Optimization (RLPO), which formulates ranking as listwise representation-level residual correction.
  arXiv  Detail & Related papers  (2026-01-12T11:45:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.