Positional Biases Shift as Inputs Approach Context Window Limits

• URL: http://arxiv.org/abs/2508.07479v1
• Date: Sun, 10 Aug 2025 20:40:24 GMT
• Title: Positional Biases Shift as Inputs Approach Context Window Limits
• Authors: Blerta Veseli, Julian Chibane, Mariya Toneva, Alexander Koller,
• Abstract summary: The LiM effect is strongest when inputs occupy up to 50% of a model's context window.<n>We observe a distance-based bias, where model performance is better when relevant information is closer to the end of the input.
• Score: 57.00239097102958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) often struggle to use information across long inputs effectively. Prior work has identified positional biases, such as the Lost in the Middle (LiM) effect, where models perform better when information appears at the beginning (primacy bias) or end (recency bias) of the input, rather than in the middle. However, long-context studies have not consistently replicated these effects, raising questions about their intensity and the conditions under which they manifest. To address this, we conducted a comprehensive analysis using relative rather than absolute input lengths, defined with respect to each model's context window. Our findings reveal that the LiM effect is strongest when inputs occupy up to 50% of a model's context window. Beyond that, the primacy bias weakens, while recency bias remains relatively stable. This effectively eliminates the LiM effect; instead, we observe a distance-based bias, where model performance is better when relevant information is closer to the end of the input. Furthermore, our results suggest that successful retrieval is a prerequisite for reasoning in LLMs, and that the observed positional biases in reasoning are largely inherited from retrieval. These insights have implications for long-context tasks, the design of future LLM benchmarks, and evaluation methodologies for LLMs handling extended inputs.

Related papers

• Context Length Alone Hurts LLM Performance Despite Perfect Retrieval [29.523005523787244]
  Large language models (LLMs) often fail to scale their performance on long-context tasks in line with the context lengths they support.<n>This paper presents findings that the answer to this question may be negative.
  arXiv  Detail & Related papers  (2025-10-06T21:17:13Z)
• Enhancing Rating Prediction with Off-the-Shelf LLMs Using In-Context User Reviews [16.394933051332657]
  Likert-scale rating prediction is a regression task that requires both language and mathematical reasoning to be solved effectively.<n>This study investigates the performance of off-the-shelf LLMs on rating prediction, providing different in-context information.<n>We find that user-written reviews significantly improve the rating prediction performance of LLMs.
  arXiv  Detail & Related papers  (2025-10-01T03:04:20Z)
• How far can bias go? -- Tracing bias from pretraining data to alignment [54.51310112013655]
  This study examines the correlation between gender-occupation bias in pre-training data and their manifestation in LLMs.<n>Our findings reveal that biases present in pre-training data are amplified in model outputs.
  arXiv  Detail & Related papers  (2024-11-28T16:20:25Z)
• On Positional Bias of Faithfulness for Long-form Summarization [83.63283027830657]
  Large Language Models (LLMs) often exhibit positional bias in long-context settings, under-attending to information in the middle of inputs.<n>We investigate the presence of this bias in long-form summarization, its impact on faithfulness, and various techniques to mitigate this bias.
  arXiv  Detail & Related papers  (2024-10-31T03:50:15Z)
• Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs [50.40165119718928]
  LongPiBench is a benchmark designed to assess positional bias involving multiple pieces of relevant information.<n>These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces.
  arXiv  Detail & Related papers  (2024-10-18T17:41:19Z)
• Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors [74.04775677110179]
  In-context Learning (ICL) has become the primary method for performing natural language tasks with Large Language Models (LLMs)<n>In this work, we examine whether this is the result of the aggregation used in corresponding datasets, where trying to combine low-agreement, disparate annotations might lead to annotation artifacts that create detrimental noise in the prompt.<n>Our results indicate that aggregation is a confounding factor in the modeling of subjective tasks, and advocate focusing on modeling individuals instead.
  arXiv  Detail & Related papers  (2024-10-17T17:16:00Z)
• Position-Aware Parameter Efficient Fine-Tuning Approach for Reducing Positional Bias in LLMs [18.832135309689736]
  Recent advances in large language models (LLMs) have enhanced their ability to process long input contexts. Recent studies show a positional bias in LLMs, demonstrating varying performance depending on the location of useful information. We develop a Position-Aware PAPEFT approach which is composed of a data augmentation technique and an efficient parameter adapter.
  arXiv  Detail & Related papers  (2024-04-01T19:04:17Z)
• Self-Supervised Position Debiasing for Large Language Models [39.261233221850155]
  We propose a self-supervised position debiasing (SOD) framework to mitigate position bias for large language models (LLMs) Experiments on eight datasets and five tasks show that SOD consistently outperforms existing methods in mitigating three types of position biases.
  arXiv  Detail & Related papers  (2024-01-02T14:12:41Z)
• Exploring the Jungle of Bias: Political Bias Attribution in Language Models via Dependency Analysis [86.49858739347412]
  Large Language Models (LLMs) have sparked intense debate regarding the prevalence of bias in these models and its mitigation. We propose a prompt-based method for the extraction of confounding and mediating attributes which contribute to the decision process. We find that the observed disparate treatment can at least in part be attributed to confounding and mitigating attributes and model misalignment.
  arXiv  Detail & Related papers  (2023-11-15T00:02:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.