Testing Retrieval-Augmented Generation Systems with Chunk Coverage

• URL: http://arxiv.org/abs/2607.18155v1
• Date: Mon, 20 Jul 2026 16:53:05 GMT
• Title: Testing Retrieval-Augmented Generation Systems with Chunk Coverage
• Abstract summary: RAG-based systems are increasingly deployed in high-stakes settings where correct behaviour depends on the retrieval component.<n>Chunk Coverage (CC) is an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems.<n>We show how CC can be used to guide test selection and generation by prioritising queries that expand coverage of previously unexercised retrieval regions.
• Score: 7.72636722205447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-Augmented Generation (RAG)-based systems\footnote{For brevity, RAG-based systems are referred to as RAG systems throughout this paper.} are increasingly deployed in high-stakes settings where correct behaviour depends not only on the language model but also on the retrieval component that selects external documents at inference time. While existing RAG evaluation metrics assess retrieval and generation quality on a per-query basis, typically relying on query-level test oracles such as reference answers or relevance annotations, they provide limited insight into whether a test suite adequately exercises the retrieval behaviour of the system as a whole. In this paper, we introduce Chunk Coverage (CC), an oracle-independent test adequacy criterion for testing the retrieval component of RAG systems. CC measures the fraction of corpus chunks that are retrieved at least once across a test suite, providing a structural view of which parts of the retrieval space have been exercised. We further show how CC can be used to guide test selection and generation by prioritising queries that expand coverage of previously unexercised retrieval regions. We evaluate CC on clinical and financial RAG system scenarios. CC-guided testing reaches 50% of attainable coverage 1.7x faster than random selection and 4.2x faster than redundancy-biased strategies. Moreover, CC improves fault detection effectiveness (APFD) by 10% to 25% over random, indicating earlier discovery of distinct retrieval faults. These results show that CC captures retrieval diversity relevant to effective testing without requiring test oracles.

Related papers

• Pre-retrieval Query Clustering for Adaptive Top-k Document Retrieval in RAG Systems [7.440012812118169]
  RAG systems commonly retrieve a fixed number of documents to ground generation.<n>simple queries suffer over-retrieval (adding noise and cost) while complex queries are under-retrieved.<n>We propose a practical, general framework for query-adaptive retrieval depth.
  arXiv  Detail & Related papers  (2026-09-11T19:57:12Z)
• Candidate-Constrained Retrieval-Augmented Generation for LongEval-RAG: System Design and Empirical Analysis [1.0312968200748118]
  We present a candidate-constrained retrieval-augmented generation system for LongEval-RAG.<n>We evaluate ten pipeline variants using a primary organizer evaluation and a supplementary self-generated diagnostic protocol.
  arXiv  Detail & Related papers  (2026-07-04T20:07:57Z)
• Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts [0.38597698994416435]
  Cluster-based semantic chunking improves retrieval and answer quality compared to fixed-size and recursion.<n>RAGAs based faithfulness shows limited reliability in this setup.
  arXiv  Detail & Related papers  (2026-07-02T08:12:35Z)
• A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation [0.0]
  Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy.<n>This paper presents a controlled empirical study of RAG sensitivity, robustness, and stability across 56 experimental runs.
  arXiv  Detail & Related papers  (2026-05-29T17:24:40Z)
• Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage [89.58253972744531]
  Retrieval-augmented generation (RAG) systems combine document retrieval with a generative model to address complex information seeking tasks like report generation.<n>We investigate whether upstream retrieval metrics can serve as reliable early indicators of the final generated response's information coverage.<n>Our findings demonstrate strong correlations between coverage-based retrieval metrics and nugget coverage in generated responses at both topic and system levels.
  arXiv  Detail & Related papers  (2026-03-09T18:20:20Z)
• Controlled Retrieval-augmented Context Evaluation for Long-form RAG [58.14561461943611]
  Retrieval-augmented generation (RAG) enhances large language models by incorporating context retrieved from external knowledge sources.<n>We argue that providing a comprehensive retrieval-augmented context is important for long-form RAG tasks like report generation.<n>We introduce CRUX, a framework designed to directly assess retrieval-augmented contexts.
  arXiv  Detail & Related papers  (2025-06-24T23:17:48Z)
• Chain-of-Retrieval Augmented Generation [91.02950964802454]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.<n>Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation [36.84847781022757]
  Retrieval-augmented generation (RAG) synergizes the retrieval of pertinent data with the generative capabilities of Large Language Models (LLMs)<n>We introduce XRAG, an open-source, modular that facilitates exhaustive evaluation of the performance of foundational components of advanced RAG modules.
  arXiv  Detail & Related papers  (2024-12-20T03:37:07Z)
• Unanswerability Evaluation for Retrieval Augmented Generation [74.3022365715597]
  UAEval4RAG is a framework designed to evaluate whether RAG systems can handle unanswerable queries effectively.<n>We define a taxonomy with six unanswerable categories, and UAEval4RAG automatically synthesizes diverse and challenging queries.
  arXiv  Detail & Related papers  (2024-12-16T19:11:55Z)
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
  Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
  arXiv  Detail & Related papers  (2024-01-30T14:25:32Z)
• Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systems [35.77217529138364]
  Several dense retrieval (DR) models have demonstrated competitive performance to term-based retrieval. DR projects queries and documents into a dense vector space and retrieves results via (approximate) nearest neighbor search. It is impossible to predict whether DR will become ubiquitous in the future, but one way this is possible is through repeated applications of decision processes.
  arXiv  Detail & Related papers  (2022-06-26T23:16:05Z)
• Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora [63.429307282665704]
  Cross-document event coreference resolution (CDCR) is an NLP task in which mentions of events need to be identified and clustered throughout a collection of documents. CDCR aims to benefit downstream multi-document applications, but improvements from applying CDCR have not been shown yet. We make the observation that every CDCR system to date was developed, trained, and tested only on a single respective corpus.
  arXiv  Detail & Related papers  (2020-11-24T17:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.