Distributionally Robust Listwise Preference Optimization

• URL: http://arxiv.org/abs/2607.01715v1
• Date: Thu, 02 Jul 2026 05:12:03 GMT
• Title: Distributionally Robust Listwise Preference Optimization
• Abstract summary: We study listwise preference optimization under ranking-label uncertainty.<n>We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list.<n> Experiments show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise.
• Score: 47.053005334474896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing robust preference optimization for language-model alignment mainly studies pairwise supervision and places robustness at the dataset, prompt, or preference-pair level. We instead study listwise preference optimization under ranking-label uncertainty: given a prompt and a candidate list, the observed ranking over that list may be ambiguous due to annotator inconsistency, near-ties, lossy rankwise feedback, or reward-model noise. We propose a pointwise total-variation robust Plackett--Luce objective that directly robustifies the ranking label conditional on the candidate list. The robust loss admits an exact decomposition into the nominal PL loss plus a worst-case PL correction, and the worst-case ranking is obtained by sorting current implicit scores in ascending order, reducing the inner maximization from $K!$ enumeration to $O(K\log K)$. This tractable structure yields strong offline and online optimization guarantees. In the offline fixed-list setting, the robust objective is convex and projected stochastic subgradient reaches global $ε$-suboptimality with $O(ε^{-2})$ sample complexity. In the online policy-induced setting, where candidate lists are generated by the current policy, we establish weak convexity and $\widetilde O(ε^{-2})$ Moreau-envelope stationarity. Experiments in offline LLM alignment show that the proposed robust correction largely preserves performance under clean labels and improves robustness under noise. In online alignment, it makes reward-model-ranked candidate expansion more reliable and improves both reward-model and external GPT-4 judge metrics.

Related papers

• Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking [2.5827686695037335]
  We study how position bias affects the ranking process induced by LLM-based rerankers.<n>We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency.
  arXiv  Detail & Related papers  (2026-08-04T04:04:12Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• Towards Order Fairness: Mitigating LLMs Order Sensitivity through Dual Group Advantage Optimization [20.259122922188126]
  Large Language Models (LLMs) suffer from order bias, where their performance is affected by the arrangement order of input elements.<n>textbfDGAO aims to improve model accuracy and order stability simultaneously.
  arXiv  Detail & Related papers  (2026-05-12T11:31:18Z)
• Talos: Optimizing Top-$K$ Accuracy in Recommender Systems [14.229918568280729]
  We propose Talos, a loss function that is specifically designed to optimize the Talos recommendation accuracy.<n>We develop a sampling-based regression algorithm for efficient and accurate threshold estimation.<n>We incorporate a tailored surrogate function to address discontinuity and enhance robustness against distribution shifts.
  arXiv  Detail & Related papers  (2026-01-27T07:04:09Z)
• RLPO: Residual Listwise Preference Optimization for Long-Context Review Ranking [50.709454968853954]
  Pointwise scoring is efficient but often fails to account for list-level interactions.<n>Listwise approaches can leverage global context, yet they are computationally expensive and become unstable as candidate lists grow.<n>We propose Residual Listwise Preference Optimization (RLPO), which formulates ranking as listwise representation-level residual correction.
  arXiv  Detail & Related papers  (2026-01-12T11:45:19Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization [78.82586283794886]
  $chi2$-Preference Optimization ($chi$PO) is an efficient offline alignment algorithm provably robust to overoptimization.<n>$chi$PO implements the principle of pessimism in the face of uncertainty via regularization.<n>$chi$PO's simplicity and strong guarantees make it the first practical and general-purpose offline alignment algorithm provably robust to overoptimization.
  arXiv  Detail & Related papers  (2024-07-18T11:08:40Z)
• Optimizing Partial Area Under the Top-k Curve: Theory and Practice [151.5072746015253]
  We develop a novel metric named partial Area Under the top-k Curve (AUTKC) AUTKC has a better discrimination ability, and its Bayes optimal score function could give a correct top-K ranking with respect to the conditional probability. We present an empirical surrogate risk minimization framework to optimize the proposed metric.
  arXiv  Detail & Related papers  (2022-09-03T11:09:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.