Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking

• URL: http://arxiv.org/abs/2608.03091v1
• Date: Tue, 04 Aug 2026 04:04:12 GMT
• Title: Position Bias Undermines Preference Consistency in Listwise LLM-Based Reranking
• Abstract summary: We study how position bias affects the ranking process induced by LLM-based rerankers.<n>We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency.
• Score: 2.5827686695037335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have emerged as promising listwise rerankers for recommender systems, but their reliability under equivalent candidate permutations remains unclear. Since recommendation candidates form an unordered set, a reranker should not depend on the arbitrary order used to serialize them. However, decoder-only LLM rerankers can allow input order to affect model scores, pairwise preferences, and rankings. We study how position bias affects the ranking process induced by LLM-based rerankers. Instead of measuring only changes in final ranked lists, we treat rankings produced under equivalent candidate permutations as observations of an induced preference system. We introduce an evaluation framework measuring pairwise preference instability, global preference inconsistency, and listwise output consistency. This framework characterizes candidate-order sensitivity at the pairwise, global, and output levels. Experiments across multiple LLMs, datasets, and list lengths show that these consistency measures are closely aligned, but can diverge from recommendation effectiveness and marginal position-exposure bias. Improving relevance or flattening exposure across positions does not necessarily restore stable pairwise preferences, globally coherent preference structures, or consistent ranked outputs. These results show that reducing marginal exposure skew is insufficient to establish ranking-function validity in LLM-based reranking. Code is available at https://github.com/ejbito/InvariRank .

Related papers

• Structure-aware Relative Policy Optimization for Ranking [35.05747484313467]
  We propose SRPO, a textbfStructure-aware textbfRelative textbfPolicy textbfOptimization framework for listwise ranking.<n>It measures the discrepancy between sampled permutations using a top-weighted Kendall-tau distance and normalizes their pairwise reward differences by the corresponding distances.<n> Experimental results across two ranking scenarios demonstrate that explicitly modeling permutation-level differences improves the effectiveness and stability of listwise ranking.
  arXiv  Detail & Related papers  (2026-07-28T04:13:47Z)
• Active Learners as Efficient PRP Rerankers [1.4892446073126189]
  Pairwise Ranking Prompting (PRP) elicits pairwise preference judgments from an LLM, which are then aggregated into a ranking, usually via classical sorting algorithms.<n>We show PRP reranking as active learning from noisy pairwise comparisons and show that active rankers are drop-in replacements that improve NDCG@10 per call in the call-constrained regime.
  arXiv  Detail & Related papers  (2026-05-14T01:03:53Z)
• F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking [79.49893545611779]
  Large Language Models (LLMs) can generate a subset and order it within a single autoregressive pass.<n>This flexibility introduces a new optimization challenge: the model must search an output space while receiving utility feedback only after the full ranked list is generated.<n>This credit assignment gap makes end-to-end optimization unstable and sample-inefficient.<n>We propose a unified framework that performs both within a single autoregressive rollout and optimize them end-to-end via factorized group-relative policy optimization.
  arXiv  Detail & Related papers  (2026-05-13T04:52:33Z)
• One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation [2.5827686695037335]
  Large language models (LLMs) are increasingly used for recommendation reranking, but their listwise predictions can depend on the order in which candidates are presented.<n>This creates a mismatch between the set-based nature of recommendation and the sequence-based computation of decoder-only LLMs.<n>We propose InvariRank, a permutation-invariant listwise reranking framework that addresses this dependence at the architectural level.
  arXiv  Detail & Related papers  (2026-04-30T08:49:44Z)
• RLPO: Residual Listwise Preference Optimization for Long-Context Review Ranking [50.709454968853954]
  Pointwise scoring is efficient but often fails to account for list-level interactions.<n>Listwise approaches can leverage global context, yet they are computationally expensive and become unstable as candidate lists grow.<n>We propose Residual Listwise Preference Optimization (RLPO), which formulates ranking as listwise representation-level residual correction.
  arXiv  Detail & Related papers  (2026-01-12T11:45:19Z)
• In-context Ranking Preference Optimization [65.5489745857577]
  We propose an In-context Ranking Preference Optimization (IRPO) framework to optimize large language models (LLMs) based on ranking lists constructed during inference.<n>We show IRPO outperforms standard DPO approaches in ranking performance, highlighting its effectiveness in aligning LLMs with direct in-context ranking preferences.
  arXiv  Detail & Related papers  (2025-04-21T23:06:12Z)
• CoRanking: Collaborative Ranking with Small and Large Ranking Agents [94.09834629572403]
  Large Language Models (LLMs) have demonstrated superior listwise ranking performance.<n>CoRanking combines small and large ranking models for efficient and effective ranking.
  arXiv  Detail & Related papers  (2025-03-30T13:00:52Z)
• LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking [17.96316956366718]
  Ranking passages by prompting a large language model (LLM) can achieve promising performance in modern information retrieval (IR) systems. We show that sorting-based methods require consistent comparisons to correctly sort the passages, which we show that LLMs often violate. We propose LLM-RankFusion, an LLM-based ranking framework that mitigates these inconsistencies and produces a robust ranking list.
  arXiv  Detail & Related papers  (2024-05-31T23:29:42Z)
• Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models [63.714662435555674]
  Large language models (LLMs) exhibit positional bias in how they use context. We propose permutation self-consistency, a form of self-consistency over ranking list outputs of black-box LLMs. Our approach improves scores from conventional inference by up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B)
  arXiv  Detail & Related papers  (2023-10-11T17:59:02Z)
• SetRank: A Setwise Bayesian Approach for Collaborative Ranking from Implicit Feedback [50.13745601531148]
  We propose a novel setwise Bayesian approach for collaborative ranking, namely SetRank, to accommodate the characteristics of implicit feedback in recommender system. Specifically, SetRank aims at maximizing the posterior probability of novel setwise preference comparisons. We also present the theoretical analysis of SetRank to show that the bound of excess risk can be proportional to $sqrtM/N$.
  arXiv  Detail & Related papers  (2020-02-23T06:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.