Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG

• URL: http://arxiv.org/abs/2605.27105v2
• Date: Wed, 27 May 2026 08:04:02 GMT
• Title: Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG
• Abstract summary: Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being input to a model's input context.<n>Prior work reports position-based effects such as lost in the middle and related long-context phenomena.<n>We show that topic sampling is a major source of variance: small topic sets can exaggerate ordering effects.<n>We also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents.
• Score: 3.597778914286147
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being concatenated into a model's input context, making both document ordering and context size critical yet controversial design choices. Prior work reports position-based effects such as lost in the middle and related long-context phenomena. However, empirical findings remain inconsistent and hard to reproduce across models, datasets, and evaluation protocols. In this paper, we present a systematic reproducibility study that revisits these claims and examines how they evolve with contemporary LLMs under a controlled evaluation framework. We first show that topic sampling is a major source of variance: small topic sets can mask or exaggerate ordering effects. Based on repeated subset sampling across multiple topic budgets, we provide a practical calibration procedure that identifies topic counts yielding stable trends at feasible cost. Using these fixed topic sets, we then reproduce and extend results on position sensitivity, re-evaluating lost in the middle and positional biases in modern LLMs. Then, we also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents. In this setting, we re-examine a recent industry study and identify discrepancies to evaluation choices such as limited topic coverage and reliance on LLM-based judges. Finally, we conduct an analysis of how retrieval order and context size affect downstream LLM performance under imperfect retrieval. Our results demonstrate that both factors interact strongly with retrieval quality and model choice, and that conclusions drawn from idealised setups do not always transfer to real-world RAG pipelines. We release all code and configurations to support reproducibility and future work on robust RAG evaluation.

Related papers

• LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal [8.33491432828288]
  We study whether query-document relevance is linearly decodable from residual-stream activations in instruction-tuned language models.<n>Our results suggest that q-d relevance is encoded as a depth-dependent signal.<n>In several models, validation-selected probes match or outperform generated judgments and better preserve system rankings.
  arXiv  Detail & Related papers  (2026-07-17T01:54:45Z)
• When Should Queries Be Decomposed? A Stage-Aware Study of Query Decomposition for Multi-Condition Retrieval [7.14438992840294]
  Multi-condition retrieval requires systems to identify documents that satisfy multiple distinct constraints.<n> query decomposition is widely adopted as an intuitive remedy, but its effectiveness across different retrieval pipeline stages remains underexplored.<n>We propose a principled Stage-Aware Decomposition framework that retains the monolithic query during initial retrieval to preserve global semantic context, while employing sub-queries exclusively during reranking for fine-grained constraint matching.
  arXiv  Detail & Related papers  (2026-06-07T11:13:15Z)
• Align Documents to Questions: Question-Oriented Document Rewriting for Retrieval-Augmented Generation [51.55755193937205]
  We propose QREAM, a style-controlled rewriter that aligns retrieved documents with a question-oriented style while preserving facts.<n>Our framework consists of two stages: (1) QREAM-ICL, which uses stylistic seeds to guide iterative rewriting exploration; and (2) QREAM-FT, a lightweight student model distilled from denoised ICL outputs.
  arXiv  Detail & Related papers  (2026-04-19T08:39:21Z)
• FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation [73.22935457705057]
  We introduce FRESCO, a benchmark for evaluating re-rankers in temporally dynamic contexts.<n>By pairing recency-seeking queries with historical Wikipedia revisions, FRESCO tests whether re-rankers can prioritize factually recent evidence while maintaining semantic relevance.<n>Our evaluation reveals a consistent failure mode across existing re-rankers: a strong bias toward older, semantically rich documents, even when they are factually obsolete.
  arXiv  Detail & Related papers  (2026-04-14T17:04:25Z)
• Dynamic Context Selection for Retrieval-Augmented Generation: Mitigating Distractors and Positional Bias [1.7674345486888503]
  Retrieval Augmented Generation (RAG) enhances language model performance by incorporating external knowledge retrieved from large corpora.<n>Standard RAG systems rely on a fixed top k retrieval strategy, which can either miss relevant information or introduce semantically irrelevant passages.<n>We propose a context-size classifier that dynamically predicts the optimal number of documents to retrieve based on query-specific informational needs.
  arXiv  Detail & Related papers  (2025-12-16T11:30:40Z)
• Towards Mixed-Modal Retrieval for Universal Retrieval-Augmented Generation [72.34977512403643]
  Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) by retrieving relevant documents from an external corpus.<n>Existing RAG systems primarily focus on unimodal text documents, and often fall short in real-world scenarios where both queries and documents may contain mixed modalities (such as text and images)<n>We propose Nyx, a unified mixed-modal to mixed-modal retriever tailored for Universal Retrieval-Augmented Generation scenarios.
  arXiv  Detail & Related papers  (2025-10-20T09:56:43Z)
• LLM-Safety Evaluations Lack Robustness [58.334290876531036]
  We argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of noise.<n>We propose a set of guidelines for reducing noise and bias in evaluations of future attack and defense papers.
  arXiv  Detail & Related papers  (2025-03-04T12:55:07Z)
• Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievals [5.605770511387228]
  RAGuard is the first benchmark to evaluate the robustness of RAG systems against misleading retrievals.<n>Unlike prior benchmarks that rely on synthetic noise, our fact-checking dataset captures naturally occurring misinformation.
  arXiv  Detail & Related papers  (2025-02-22T05:50:15Z)
• RAGEval: Scenario Specific RAG Evaluation Dataset Generation Framework [66.93260816493553]
  This paper introduces RAGEval, a framework designed to assess RAG systems across diverse scenarios.<n>With a focus on factual accuracy, we propose three novel metrics: Completeness, Hallucination, and Irrelevance.<n> Experimental results show that RAGEval outperforms zero-shot and one-shot methods in terms of clarity, safety, conformity, and richness of generated samples.
  arXiv  Detail & Related papers  (2024-08-02T13:35:11Z)
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
  Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
  arXiv  Detail & Related papers  (2024-01-30T14:25:32Z)
• The Power of Noise: Redefining Retrieval for RAG Systems [19.387105120040157]
  Retrieval-Augmented Generation (RAG) has emerged as a method to extend beyond the pre-trained knowledge of Large Language Models. We focus on the type of passages IR systems within a RAG solution should retrieve.
  arXiv  Detail & Related papers  (2024-01-26T14:14:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.