Is Position Bias in Dense Retrievers Built In-or Learned from Data?

• URL: http://arxiv.org/abs/2605.26578v1
• Date: Tue, 26 May 2026 05:50:16 GMT
• Title: Is Position Bias in Dense Retrievers Built In-or Learned from Data?
• Abstract summary: We study how the positional distribution of evidence in training data affects retrieval-level bias direction.<n>Position-balanced training reduces positional sensitivity by 57--87% on position-aware benchmarks.<n>These results identify training-position distribution as a major controllable factor in retrieval-level position bias.
• Score: 3.331218753253791
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dense retrievers exhibit positional bias, favoring documents whose query-relevant information appears near the beginning and degrading retrieval performance when the information appears later. While prior work on positional bias in dense retrievers has largely focused on architectural explanations, we study how the positional distribution of evidence in training data affects retrieval-level bias direction. To test this, we construct synthetic position-targeted training sets in which query-relevant evidence appears at the beginning, middle, or end of documents, and fine-tune eight architecturally diverse pretrained models under position-skewed and balanced training distributions. At the ranking level, we observe a strong directional pattern across the examined models: skewed training distributions favor evidence at the corresponding positions. Position-balanced training reduces positional sensitivity by 57--87\% on position-aware benchmarks, with competitive mean retrieval performance in our controlled setting. Representation-level analyses further suggest that fine-tuning often reshapes learned positional preferences, although pre-existing architectural or pretraining-specific tendencies persist in some models. These results identify training-position distribution as a major controllable factor in retrieval-level position bias and suggest balanced data curation as a practical mitigation strategy.

Related papers

• LLM-based Listwise Reranking under the Effect of Positional Bias [69.07550988151473]
  LLM-based listwise passage reranking has attracted attention for its effectiveness in ranking candidate passages.<n>These models suffer from positional bias, where passages positioned towards the end of the input are less likely to be moved to top positions in the ranking.<n>We propose DebiasFirst, a method that integrates positional calibration and position-aware data augmentation during fine-tuning.
  arXiv  Detail & Related papers  (2026-04-04T08:40:57Z)
• PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark [12.848308213591622]
  PosIR (Position-Aware Information Retrieval) is a comprehensive benchmark designed to diagnose position bias in diverse retrieval scenarios.<n>PosIR comprises 310 datasets spanning 10 languages and 31 domains, constructed through a rigorous pipeline that ties relevance to precise reference spans.
  arXiv  Detail & Related papers  (2026-01-13T09:22:16Z)
• Layer-wise Positional Bias in Short-Context Language Modeling [5.417332705560665]
  We introduce an attribution-based framework to analyze positional effects in short-context language modeling.<n>We quantify how each layer distributes importance across input positions, yielding layer-wise positional importance profiles.<n>Characterizing these profiles, we find prominent recency bias that increases with depth and subtle primacy bias that diminishes through model depth.
  arXiv  Detail & Related papers  (2026-01-07T17:04:30Z)
• An Empirical Study of Position Bias in Modern Information Retrieval [9.958646803388513]
  This study investigates the position bias in information retrieval.<n>Models tend to overemphasize content at the beginning of passages while neglecting semantically relevant information that appears later.<n>Experiments show that when relevant information appears later in the passage, dense embedding models and ColBERT-style models suffer significant performance degradation.
  arXiv  Detail & Related papers  (2025-05-20T05:29:01Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Referee-Meta-Learning for Fast Adaptation of Locational Fairness [26.770426062329165]
  We propose a locational meta-referee (Meta-Ref) to oversee the few-shot meta-training and meta-testing of a deep neural network. We show that Meta-Ref can improve locational fairness while keeping the overall prediction quality at a similar level.
  arXiv  Detail & Related papers  (2024-02-20T21:09:04Z)
• A Frustratingly Easy Improvement for Position Embeddings via Random Padding [68.75670223005716]
  In this paper, we propose a simple but effective strategy, Random Padding, without any modifications to existing pre-trained language models. Experiments show that Random Padding can significantly improve model performance on the instances whose answers are located at rear positions.
  arXiv  Detail & Related papers  (2023-05-08T17:08:14Z)
• Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics [88.39382177059747]
  A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution. Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2023-02-03T02:40:54Z)
• General Greedy De-bias Learning [163.65789778416172]
  We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space. GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
  arXiv  Detail & Related papers  (2021-12-20T14:47:32Z)
• Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice [52.11183787786718]
  Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications. Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy. We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
  arXiv  Detail & Related papers  (2021-11-24T06:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.