Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions

• URL: http://arxiv.org/abs/2606.06758v2
• Date: Mon, 08 Jun 2026 19:53:29 GMT
• Title: Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions
• Abstract summary: A model may answer from parametric priors, fail to use evidence that is present, or cite relevant text without converting it into the final answer.<n>This paper introduces a four-condition diagnostic protocol for evidence-utilization evaluation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Final-answer accuracy, retrieval recall, and citation overlap do not reveal how much answer advantage a long-context or retrieval-augmented language model actually recovers from supplied evidence. A model may answer from parametric priors, fail to use evidence that is present, or cite relevant text without converting it into the final answer. This paper introduces a four-condition diagnostic protocol for evidence-utilization evaluation under matched examples, models, prompts, and scoring rules. The protocol compares no-evidence, full-context, retrieved-evidence, and oracle-evidence reference conditions, and uses Oracle-Reference Normalized Context Utilization (ONCU) as a denominator-valid estimate of recovered oracle-reference evidence advantage. The empirical study evaluates five local open-weight models from the Qwen, Gemma, Llama, and Mistral families over Controlled-ONCU-safe16K, HotpotQA-ONCU, and 2WikiMultiHopQA-ONCU, comprising 18,000 ONCU-compatible predictions. Results show a task-dependent diagnostic pattern: controlled synthetic settings expose reduced recovery when the same evidence is embedded in long input rather than supplied compactly, while realistic multi-hop reconstructions show that full-context inputs outperform the tested retrieved inputs in denominator-free answer and evidence metrics, with ONCU supporting the same direction on oracle-improving groups. Sensitivity audits with stronger retrieval settings narrow some gaps but do not overturn the scoped interpretation. The main contribution is therefore not a single utilization ratio, but a matched diagnostic protocol that separates no-evidence answerability, oracle-evidence recoverability, full-context recovery, retrieval-conditioned recovery, denominator validity, and companion answer/evidence diagnostics.

Related papers

• Beyond Relevance: Bayesian Evidence Acquisition for Agentic Whole-Slide Image Reasoning [72.86819994769634]
  Whole-slide image (WSI) reasoning requires an agent to sequentially acquire visual evidence before answering a diagnostic question.<n>We propose BEACON, a plug-and-play agentic framework that reformulates WSI reasoning as a Bayesian evidence acquisition problem.
  arXiv  Detail & Related papers  (2026-08-06T08:40:57Z)
• HiEviDR-Bench: A Benchmark for Hierarchical Evidence Aggregation in Deep Research [52.87128503345243]
  We introduce HiEviDR-Bench, a benchmark for evaluating Hierarchical Evidence Aggregation in Deep Research.<n>HiEviDR-Bench represents each instance with an explicit evidence graph that captures evidence selection, cross-source linking, and aggregation.<n>We develop a traceability-oriented evaluation framework with five dimensions: report quality, evidence traceability, citation accuracy, claim verification, and answer correctness.
  arXiv  Detail & Related papers  (2026-07-27T23:50:46Z)
• Before the Action: Benchmarking LLMs on Prospective Hypothesis Discovery [95.44432473367836]
  Large language models (LLMs) excel at answering pre-specified questions, yet their ability to navigate the open-ended, pre-conclusion stage of discovery remains largely unmeasured.<n>We introduce Prospective Hypothesis Discovery (PHD), which asks models to autonomously construct grounded, discnative, and testable hypothesis spaces from inconclusive evidence.<n>We introduce HypoArena, comprising HypoData, a benchmark of 988 cases across six scientific and analytical domains, and HypoEval, an evaluation framework for open-ended hypothesis sets.
  arXiv  Detail & Related papers  (2026-07-17T08:56:43Z)
• Beyond the Leaderboard: Design Lessons for Trustworthy Multimodal VQA [4.080704298118247]
  multimodal AI must combine visual and textual evidence while remaining reliable and interpretable.<n>We analyze design choices across nine documented systems for question answering and explanation quality.<n>The findings support trustworthy multimodal healthcare AI based on data fusion, explainability, and resilient evaluation.
  arXiv  Detail & Related papers  (2026-07-16T17:38:19Z)
• AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora [6.956097396264084]
  We introduce AuthTrace, a diagnostic benchmark built on thematically dense single-author corpora.<n>AuthTrace provides explicit quoted evidence, exact fan-in annotation, and a unified pack-level protocol measuring evidence recall, evidence precision, and answer correctness.
  arXiv  Detail & Related papers  (2026-05-25T03:10:52Z)
• DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding [29.270128057512284]
  We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem.<n>We design a four-stage evaluation protocol that audits each level of the trajectory independently.<n>We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems.
  arXiv  Detail & Related papers  (2026-05-09T11:12:59Z)
• Diagnosable ColBERT: Debugging Late-Interaction Retrieval Models Using a Learned Latent Space as Reference [1.6904475483445454]
  We propose a framework that aligns ColBERT token embeddings to a reference latent space grounded in clinical knowledge.<n>This alignment turns document encodings into inspectable evidence of what the model appears to understand.
  arXiv  Detail & Related papers  (2026-04-21T15:19:03Z)
• Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG [8.18791900871137]
  Retrieval-Augmented Generation aims to reduce hallucinations by grounding answers in retrieved evidence.<n>Existing evaluations focus on answer-level or passage-level accuracy, offering limited insight into how evidence is used during generation.<n>We introduce a facet-level diagnostics framework for QA that decomposes each input question into atomic reasoning facets.
  arXiv  Detail & Related papers  (2026-04-10T09:59:43Z)
• PAVE: Premise-Aware Validation and Editing for Retrieval-Augmented LLMs [8.082352336629816]
  PAVE is an inference-time validation layer for evidence-grounded question answering.<n>It decomposes retrieved context into question-conditioned atomic facts, drafts an answer, scores how well that draft is supported by the extracted premises, and revises low-support outputs before finalization.
  arXiv  Detail & Related papers  (2026-03-21T06:23:21Z)
• When Iterative RAG Beats Ideal Evidence: A Diagnostic Study in Scientific Multi-hop Question Answering [0.2796197251957245]
  We study whether synchronized iterative retrieval and reasoning can surpass an idealized static upper bound (Gold Context) RAG.<n>We benchmark eleven state-of-the-art LLMs under three regimes: (i) No Context, measuring reliance on parametric memory; (ii) Gold Context, where all evidence is supplied at once; and (iii) Iterative RAG, a training-free oracle controller that alternates retrieval, hypothesis refinement, and evidence-aware stopping.<n>Across models, Iterative RAG consistently outperforms Gold Context, with gains up to 25.6 percentage points, especially for non-reasoning
  arXiv  Detail & Related papers  (2026-01-27T17:35:05Z)
• Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation [108.13261761812517]
  We introduce FRANQ (Faithfulness-based Retrieval Augmented UNcertainty Quantification), a novel method for hallucination detection in RAG outputs.<n>We present a new long-form Question Answering (QA) dataset annotated for both factuality and faithfulness.
  arXiv  Detail & Related papers  (2025-05-27T11:56:59Z)
• Unstructured Evidence Attribution for Long Context Query Focused Summarization [53.08341620504465]
  We propose to extract unstructured (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case.<n>We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle"
  arXiv  Detail & Related papers  (2025-02-20T09:57:42Z)
• Contrastive Learning to Improve Retrieval for Real-world Fact Checking [84.57583869042791]
  We present Contrastive Fact-Checking Reranker (CFR), an improved retriever for fact-checking complex claims. We leverage the AVeriTeC dataset, which annotates subquestions for claims with human written answers from evidence documents. We find a 6% improvement in veracity classification accuracy on the dataset.
  arXiv  Detail & Related papers  (2024-10-07T00:09:50Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• From Relevance to Utility: Evidence Retrieval with Feedback for Fact Verification [118.03466985807331]
  We argue that, rather than relevance, for FV we need to focus on the utility that a claim verifier derives from the retrieved evidence.<n>We introduce the feedback-based evidence retriever(FER) that optimize the evidence retrieval process by incorporating feedback from the claim verifier.
  arXiv  Detail & Related papers  (2023-10-18T02:59:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.