Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation

• URL: http://arxiv.org/abs/2609.15800v1
• Date: Mon, 14 Sep 2026 16:11:35 GMT
• Title: Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation
• Abstract summary: We propose a unified agent loop for explicit evidence selection and consolidation.<n>During exploration, SCoRE retains only query-relevant observations and their source pointers in a maintained textual ledger.<n>At termination, it reloads the referenced original images and consolidates the visual evidence for answering, arranging it into a logical sequence.
• Score: 50.92223196290564
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual Retrieval-Augmented Generation (VRAG) empowers models to navigate and answer queries about visually rich documents by retrieving relevant page images as visual evidence and reasoning over their content. However, effectively utilizing this visual evidence is usually impeded by two main challenges. First, answer-relevant evidence is sparse and may be concentrated in a small region of one page or dispersed across multiple pages. Second, existing agentic methods often generate answers based on raw exploration trajectories or compressed textual memories rather than an explicitly organized set of supporting images, making answers susceptible to exploration noise and obscuring the evidence-backed reasoning trace. We argue that the bottleneck lies not only in evidence discovery but also in its preservation and organization before answer generation. We propose SCoRE (Selection and Consolidation for Robust Evidence), a unified agent loop for explicit evidence selection and consolidation. During exploration, SCoRE retains only query-relevant observations and their source pointers in a maintained textual ledger, preserving earlier evidence while keeping the visual context bounded. At termination, it reloads the referenced original images and consolidates the visual evidence for answering, arranging it into a logical sequence. This decouples final reasoning from exploratory trial-and-error while ensuring strict visual grounding via indexed claim-to-image linkages. To enable end-to-end optimization of this unified rollout, our training paradigm combines filtered cold-start trajectory distillation with evidence-aware reinforcement learning, whose reward promotes evidence coverage, consolidation compactness, and answer correctness.

Related papers

• Beyond One-Shot Expansion: Contrastive Evidence Exploration for Multi-Hop Retrieval [19.37201042661162]
  We propose a training-free multi-hop retrieval framework that integrates evidence-conditioned exploration, passage-specific contrastive refinement, and coverage-aware final ranking.<n> Experiments on MuSiQue, HotpotQA, and 2WikiMultihopQA demonstrate consistent improvements in retrieval quality and downstream QA performance over baselines.
  arXiv  Detail & Related papers  (2026-09-07T05:13:56Z)
• SAFE-G: Structure-aware Faithful Evidence-guided Generation for Knowledge-based Visual Question Answering [8.538563236020947]
  SAFE-G is a structure-Aware Faithful Evidence-guided Generation framework.<n>It enables precise evidence localization and trustworthy reasoning.<n>It outperforms prior methods by a margin of 8.9% and 3.5%.
  arXiv  Detail & Related papers  (2026-08-22T06:34:53Z)
• GLaQ: Grounding Latent Queries in Visual Evidence for Multimodal Reasoning [27.232897918541067]
  Chain-of-thought reasoning has substantially improved the problem-solving capabilities of multimodal large language models.<n>Fine-grained visual evidence, however, remains difficult to preserve and reuse across text-based reasoning steps.<n>We introduce GLaQ, a grounded latent-query framework that replaces sequential latent rollout with a fixed set of context-conditioned queries.
  arXiv  Detail & Related papers  (2026-08-16T04:01:08Z)
• REVEAL: A Rubric-Guided Agent for Explicit Evidence Sufficiency Verificationin Long-Video Question Answering [25.519927761448603]
  REVEAL is a rubric-guided agent framework for long-video question answering.<n>We introduce an adaptive visual-similarity-based preprocessing pipeline that groups visually coherent adjacent frames into natural event units.<n>Built upon this structured memory, REVEAL uses an automatically constructed rubric library to explicitly verify whether retrieved evidence satisfies sufficiency criteria.
  arXiv  Detail & Related papers  (2026-08-09T09:55:42Z)
• DRNOISE: Benchmarking Deep Research Agents in Misleading Evidence Environments [51.049999642138]
  We introduce DRNOISE, a 100-task benchmark for answer recovery under misleading evidence.<n>Each task has a unique gold answer supported by two corroborating indirect record chains; the paired noisy condition adds one plausible document that states a conflicting answer directly.<n>Across agents with strong clean-task performance, this single intervention causes 66-88 percentage-point accuracy drops.
  arXiv  Detail & Related papers  (2026-07-19T15:20:40Z)
• REVERSE: Reinforcing Evidence Verification and Search for Agentic Image geo-localization [15.44857530497746]
  We present REVERSE, a framework that teaches three intermediate decisions: where to look, what to query, and what evidence to trust.<n>An offline search cache makes retrieval observations stable and reusable during reinforcement learning, enabling dense supervision over noisy search results.<n>With a 4B model, REVERSE outperforms strong retrieval-augmented baselines and rivals substantially larger models on Im2GPS3k and YFCC4k.
  arXiv  Detail & Related papers  (2026-05-26T11:20:09Z)
• Look As You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning [55.232400251303794]
  Look As You Think (LAT) is a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution.<n>LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in IoU@0.5.
  arXiv  Detail & Related papers  (2025-11-15T02:50:23Z)
• VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification [107.75781898355562]
  We introduce a novel framework, called VeriCite, designed to rigorously validate supporting evidence and enhance answer attribution.<n>We conduct experiments across five open-source LLMs and four datasets, demonstrating that VeriCite can significantly improve citation quality while maintaining the correctness of the answers.
  arXiv  Detail & Related papers  (2025-10-13T13:38:54Z)
• VisRAG 2.0: Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation [64.82775032985485]
  Visual retrieval-augmented generation (VRAG) augments vision-language models (VLMs) with external visual knowledge to ground reasoning and reduce hallucinations.<n>Yet current VRAG systems often fail to reliably perceive and integrate evidence across multiple images, leading to weak grounding and erroneous conclusions.<n>We propose EVisRAG, an end-to-end framework that learns to reason with evidence-guided multi-image to address this issue.
  arXiv  Detail & Related papers  (2025-10-10T13:34:23Z)
• ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generation [82.54090885503287]
  Retrieval-Augmented Generation augments Large Language Models with external knowledge to improve factuality.<n>Existing RAG systems fail to extract and integrate the key clues needed to support faithful and interpretable reasoning.<n>We propose ClueAnchor, a novel framework for enhancing RAG via clue-anchored reasoning exploration and optimization.
  arXiv  Detail & Related papers  (2025-05-30T09:18:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.