Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation

• URL: http://arxiv.org/abs/2604.20763v1
• Date: Wed, 22 Apr 2026 16:49:30 GMT
• Title: Coverage, Not Averages: Semantic Stratification for Trustworthy Retrieval Evaluation
• Abstract summary: Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG)<n>We formalize evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction.<n>This yields (1) formal coverage guarantees across retrieval semantic regimes and (2) interpretable visibility into retrieval failure modes.
• Score: 3.4588983513116065
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval quality is the primary bottleneck for accuracy and robustness in retrieval-augmented generation (RAG). Current evaluation relies on heuristically constructed query sets, which introduce a hidden intrinsic bias. We formalize retrieval evaluation as a statistical estimation problem, showing that metric reliability is fundamentally limited by the evaluation-set construction. We further introduce \emph{semantic stratification}, which grounds evaluation in corpus structure by organizing documents into an interpretable global space of entity-based clusters and systematically generating queries for missing strata. This yields (1) formal semantic coverage guarantees across retrieval regimes and (2) interpretable visibility into retrieval failure modes. Experiments across multiple benchmarks and retrieval methods validate our framework. The results expose systematic coverage gaps, identify structural signals that explain variance in retrieval performance, and show that stratified evaluation yields more stable and transparent assessments while supporting more trustworthy decision-making than aggregate metrics.

Related papers

• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization [5.106530060248491]
  LongSumEval is a unified framework bridging evaluation and generation through structured question-answering feedback.<n>The framework operationalizes summary quality as answerability and factual alignment of question-answer pairs.<n> Structured feedback enables significant quality improvements through self-refinement without retraining.
  arXiv  Detail & Related papers  (2026-04-28T02:07:56Z)
• K$α$LOS finds Consensus: A Meta-Algorithm for Evaluating Inter-Annotator Agreement in Complex Vision Tasks [4.297070083645049]
  We propose K$$LOS, a unified meta-algorithm that generalizes the "Localization First" principle to dataset quality evaluation.<n>By resolving spatial correspondence before assessing agreement, our framework transforms complex-categorical problems into nominal reliability.
  arXiv  Detail & Related papers  (2026-03-28T08:54:05Z)
• DREAM: Deep Research Evaluation with Agentic Metrics [21.555357444628044]
  We propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that makes evaluation itself agentic.<n> DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent.<n>Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks.
  arXiv  Detail & Related papers  (2026-02-21T19:14:31Z)
• TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing [10.736029638634504]
  We formalize topological consistency as the fundamental property of topological maps and show that localization accuracy provides an efficient surrogate metric.<n>We propose the first quantitative measure of dataset ambiguity to enable fair comparisons across environments.<n>All datasets, baselines, and evaluation tools are fully open-sourced to foster consistent and reproducible research in topological mapping.
  arXiv  Detail & Related papers  (2025-10-05T08:58:08Z)
• SCORE: A Semantic Evaluation Framework for Generative Document Parsing [2.5101597298392098]
  Multi-modal generative document parsing systems produce semantically correct yet structurally divergent outputs.<n>Conventional metrics-CER, WER, IoU, or TEDS-misclassify such diversity as error, penalizing valid interpretations and obscuring system behavior.<n>We introduce SCORE, an interpretation-agnostic framework that integrates (i) adjusted edit distance for robust content fidelity, (ii) token-level diagnostics to distinguish hallucinations from omissions, (iii) table evaluation with spatial tolerance and semantic alignment, and (iv) hierarchy-aware consistency checks.
  arXiv  Detail & Related papers  (2025-09-16T16:06:19Z)
• CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection [56.302586730134806]
  We introduce Confidence-Consistency Evaluation (CCE), a novel evaluation metric.<n>CCE simultaneously measures prediction confidence and uncertainty consistency.<n>We also establish RankEval, a benchmark for comparing the ranking capabilities of various metrics.
  arXiv  Detail & Related papers  (2025-09-01T03:38:38Z)
• Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluation [0.0]
  Vector retrieval systems exhibit significant performance variance across queries due to heterogeneous embedding quality.<n>We propose a framework for predicting retrieval performance at the query level by combining quantization robustness and neighborhood density metrics.
  arXiv  Detail & Related papers  (2025-07-08T12:33:11Z)
• Aurora: Are Android Malware Classifiers Reliable and Stable under Distribution Shift? [51.12297424766236]
  AURORA is a framework to evaluate malware classifiers based on their confidence quality and operational resilience.<n>AURORA is complemented by a set of metrics designed to go beyond point-in-time performance.<n>The fragility in SOTA frameworks across datasets of varying drift suggests the need for a return to the whiteboard.
  arXiv  Detail & Related papers  (2025-05-28T20:22:43Z)
• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection.<n>Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains.<n>We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• A Backdoor-based Explainable AI Benchmark for High Fidelity Evaluation of Attributions [60.06461883533697]
  We first identify a set of fidelity criteria that reliable benchmarks for attribution methods are expected to fulfill.<n>We then introduce a Backdoor-based eXplainable AI benchmark (BackX) that adheres to the desired fidelity criteria.<n>Our analysis also offers insights into defending against neural Trojans by utilizing the attributions.
  arXiv  Detail & Related papers  (2024-05-02T13:48:37Z)
• A structured regression approach for evaluating model performance across intersectional subgroups [53.91682617836498]
  Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups. We introduce a structured regression approach to disaggregated evaluation that we demonstrate can yield reliable system performance estimates even for very small subgroups.
  arXiv  Detail & Related papers  (2024-01-26T14:21:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.