Structure-aware Relative Policy Optimization for Ranking

• URL: http://arxiv.org/abs/2607.25268v1
• Date: Tue, 28 Jul 2026 04:13:47 GMT
• Title: Structure-aware Relative Policy Optimization for Ranking
• Abstract summary: We propose SRPO, a textbfStructure-aware textbfRelative textbfPolicy textbfOptimization framework for listwise ranking.<n>It measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances.<n> Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking.
• Score: 35.05747484313467
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Ranking is a fundamental component of modern information access systems. Reinforcement learning (RL) provides a flexible framework for directly optimizing coarse-grained feedback and system-level objectives defined over the complete ranking list. However, existing RL-based ranking methods typically treat each sampled permutation as an atomic output and evaluate it primarily through a scalar reward, overlooking the structural relationships among different ranking lists. Consequently, permutations with similar rewards but substantially different permutation patterns may receive comparable optimization signals, potentially leading to inaccurate credit assignment and overly aggressive policy updates. To address this limitation, we propose SRPO, a \textbf{S}tructure-aware \textbf{R}elative \textbf{P}olicy \textbf{O}ptimization framework for listwise ranking. SRPO measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances. It quantifies the reward improvement per unit of ranking change, thereby emphasizing efficient local refinements, particularly those involving top-ranked positions. Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking, with particularly favorable performance in limited-feedback and complex list-level optimization settings.

Related papers

• Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking [2.5827686695037335]
  We study how position bias affects the ranking process induced by LLM-based rerankers.<n>We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency.
  arXiv  Detail & Related papers  (2026-08-04T04:04:12Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation [2.5827686695037335]
  Large language models (LLMs) are increasingly used for recommendation reranking, but their listwise predictions can depend on the order in which candidates are presented.<n>This creates a mismatch between the set-based nature of recommendation and the sequence-based computation of decoder-only LLMs.<n>We propose InvariRank, a permutation-invariant listwise reranking framework that addresses this dependence at the architectural level.
  arXiv  Detail & Related papers  (2026-04-30T08:49:44Z)
• Reward Score Matching: Unifying Reward-based Fine-tuning for Flow and Diffusion Models [54.597200388738656]
  Reward-based fine-tuning aims to steer a pretrained diffusion or flow-based generative model toward higher-reward samples.<n>We show that many can be written under a common framework, which we call reward score matching (RSM)
  arXiv  Detail & Related papers  (2026-04-19T12:47:52Z)
• RankList -- A Listwise Preference Learning Framework for Predicting Subjective Preferences [66.76322360727809]
  We propose RankList, a listwise preference learning framework that generalizes RankNet to structured list-level supervision.<n>Our formulation explicitly models local and non-local ranking constraints within a probabilistic framework.<n>Extensive experiments demonstrate the superiority of our method across diverse modalities.
  arXiv  Detail & Related papers  (2025-08-13T13:59:41Z)
• In-context Ranking Preference Optimization [65.5489745857577]
  We propose an In-context Ranking Preference Optimization (IRPO) framework to optimize large language models (LLMs) based on ranking lists constructed during inference.<n>We show IRPO outperforms standard DPO approaches in ranking performance, highlighting its effectiveness in aligning LLMs with direct in-context ranking preferences.
  arXiv  Detail & Related papers  (2025-04-21T23:06:12Z)
• Learning k-Determinantal Point Processes for Personalized Ranking [13.677246792673564]
  We present a new optimization criterion LkP based on set probability comparison for personalized ranking. LkP is broadly applicable, and when applied to existing recommendation models it also yields strong performance improvements.
  arXiv  Detail & Related papers  (2024-06-23T02:24:50Z)
• Probabilistic Permutation Graph Search: Black-Box Optimization for Fairness in Ranking [53.94413894017409]
  We present a novel way of representing permutation distributions, based on the notion of permutation graphs. Similar to PL, our distribution representation, called PPG, can be used for black-box optimization of fairness.
  arXiv  Detail & Related papers  (2022-04-28T20:38:34Z)
• SetRank: A Setwise Bayesian Approach for Collaborative Ranking from Implicit Feedback [50.13745601531148]
  We propose a novel setwise Bayesian approach for collaborative ranking, namely SetRank, to accommodate the characteristics of implicit feedback in recommender system. Specifically, SetRank aims at maximizing the posterior probability of novel setwise preference comparisons. We also present the theoretical analysis of SetRank to show that the bound of excess risk can be proportional to $sqrtM/N$.
  arXiv  Detail & Related papers  (2020-02-23T06:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.