Listwise Preference Alignment Optimization for Tail Item Recommendation

• URL: http://arxiv.org/abs/2507.02255v1
• Date: Thu, 03 Jul 2025 03:08:23 GMT
• Title: Listwise Preference Alignment Optimization for Tail Item Recommendation
• Authors: Zihao Li, Chao Yang, Tong Zhang, Yakun Chen, Xianzhi Wang, Guandong Xu, Daoyi Dong,
• Abstract summary: We propose LPO4Rec, which extends the Bradley-Terry model from pairwise comparison to listwise comparison.<n>Specifically, we derive a closed form optimal policy to enable more efficient and effective training without explicit reward modeling.<n>Our experiments on three public datasets show that our method outperforms 10 baselines by a large margin.
• Score: 29.20780001894192
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Preference alignment has achieved greater success on Large Language Models (LLMs) and drawn broad interest in recommendation research. Existing preference alignment methods for recommendation either require explicit reward modeling or only support pairwise preference comparison. The former directly increases substantial computational costs, while the latter hinders training efficiency on negative samples. Moreover, no existing effort has explored preference alignment solutions for tail-item recommendation. To bridge the above gaps, we propose LPO4Rec, which extends the Bradley-Terry model from pairwise comparison to listwise comparison, to improve the efficiency of model training. Specifically, we derive a closed form optimal policy to enable more efficient and effective training without explicit reward modeling. We also present an adaptive negative sampling and reweighting strategy to prioritize tail items during optimization and enhance performance in tail-item recommendations. Besides, we theoretically prove that optimizing the listwise preference optimization (LPO) loss is equivalent to maximizing the upper bound of the optimal reward. Our experiments on three public datasets show that our method outperforms 10 baselines by a large margin, achieving up to 50% performance improvement while reducing 17.9% GPU memory usage when compared with direct preference optimization (DPO) in tail-item recommendation. Our code is available at https://github.com/Yuhanleeee/LPO4Rec.

Related papers

• Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization [66.67988187816185]
  We aim to emphscale up the number of on-policy samples via repeated random sampling to improve alignment performance.<n>Our experiments reveal that this strategy leads to a emphdecline in performance as the sample size increases.<n>We introduce a scalable preference data construction strategy that consistently enhances model performance as the sample scale increases.
  arXiv  Detail & Related papers  (2025-02-24T04:22:57Z)
• Calibrated Multi-Preference Optimization for Aligning Diffusion Models [92.90660301195396]
  Calibrated Preference Optimization (CaPO) is a novel method to align text-to-image (T2I) diffusion models.<n>CaPO incorporates the general preference from multiple reward models without human annotated data.<n> Experimental results show that CaPO consistently outperforms prior methods.
  arXiv  Detail & Related papers  (2025-02-04T18:59:23Z)
• Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment [19.02679077706812]
  We study the problem of aligning large language models with human preference data.<n>We propose direct preference optimization (Cal-DPO), a simple yet effective algorithm.<n>The results of our experiments on a variety of standard benchmarks show that Cal-DPO remarkably improves off-the-shelf methods.
  arXiv  Detail & Related papers  (2024-12-19T04:31:56Z)
• Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts [17.243429150450886]
  We propose $textbfMulti-Preference Optimization (MPO) to optimize over entire sets of responses.<n>MPO employs deviation-based weighting, which emphasizes outlier responses that deviate most from the mean reward.<n>We theoretically prove that MPO reduces alignment bias at a rate of $mathcalOleft(frac1sqrtnright)$ with respect to the number of responses per query.
  arXiv  Detail & Related papers  (2024-12-05T21:50:22Z)
• Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization [14.50339880957898]
  We aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced.
  arXiv  Detail & Related papers  (2024-11-07T23:03:11Z)
• Ordinal Preference Optimization: Aligning Human Preferences via NDCG [28.745322441961438]
  We develop an end-to-end preference optimization algorithm by approxing NDCG with a differentiable surrogate loss. OPO outperforms existing pairwise and listwise approaches on evaluation sets and general benchmarks like AlpacaEval.
  arXiv  Detail & Related papers  (2024-10-06T03:49:28Z)
• AIPO: Improving Training Objective for Iterative Preference Optimization [34.24211649396053]
  We study iterative preference optimization with synthetic data. We propose our training objective for iterative preference optimization, namely Agreement-aware Iterative Preference Optimization (AIPO)
  arXiv  Detail & Related papers  (2024-09-13T14:03:49Z)
• On Softmax Direct Preference Optimization for Recommendation [50.896117978746]
  We propose Softmax-DPO (S-DPO) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders.
  arXiv  Detail & Related papers  (2024-06-13T15:16:11Z)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
  Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values. We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO) Our method is versatile and can be readily adapted to various preference optimization frameworks.
  arXiv  Detail & Related papers  (2024-06-04T20:33:22Z)
• Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization [105.3612692153615]
  We propose a new axis based on eliciting preferences jointly over instruction-response pairs.<n>Joint preferences over instruction and response pairs can significantly enhance the alignment of large language models.
  arXiv  Detail & Related papers  (2024-03-31T02:05:40Z)
• Towards Efficient Exact Optimization of Language Model Alignment [93.39181634597877]
  Direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. We show that DPO derived based on the optimal solution of problem leads to a compromised mean-seeking approximation of the optimal solution in practice. We propose efficient exact optimization (EXO) of the alignment objective.
  arXiv  Detail & Related papers  (2024-02-01T18:51:54Z)
• Reward Model Ensembles Help Mitigate Overoptimization [7.715463015544845]
  Reinforcement learning from human feedback (RLHF) is a standard approach for fine-tuning large language models to follow instructions. As imperfect representations of the "true" reward, learned reward models are susceptible to overoptimization.
  arXiv  Detail & Related papers  (2023-10-04T11:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.