ViSAR: Training-Free Adaptive-$k$ Retrieval for Visual Document Question Answering

• URL: http://arxiv.org/abs/2609.02486v2
• Date: Thu, 03 Sep 2026 10:07:37 GMT
• Title: ViSAR: Training-Free Adaptive-$k$ Retrieval for Visual Document Question Answering
• Abstract summary: ViSAR (Visual Semantic Activation Retrieval) is a training-free adaptive-$k$ retrieval method for late-interaction visual document retrieval.<n>We show that ViSAR retrieves query-adapted page sets that reduce RAG latency by up to 58.7%, while maintaining or improving answer accuracy.
• Score: 0.7481381091903384
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Document Visual Question Answering (DocVQA) often leverages Retrieval-Augmented Generation (RAG), where late-interaction encoders are commonly used to identify document pages relevant to a user query, before answer generation by a Large Vision-Language Model (LVLM). Existing approaches typically retrieve a fixed top-$k$ number of pages regardless of query complexity, which increases LVLM latency and may degrade answer accuracy. We introduce ViSAR (Visual Semantic Activation Retrieval), a training-free adaptive-$k$ retrieval method for late-interaction visual document retrieval. ViSAR operates directly in the embedding space to construct a query-conditioned page-level similarity matrix that highlights query-relevant semantics and dynamically determines the number of pages to retrieve. Across multiple encoders and LVLMs, ViSAR retrieves compact, query-adapted page sets that reduce RAG latency by up to 58.7\%, while maintaining or improving answer accuracy compared with fixed top-$k$ and adaptive retrieval heuristics. Furthermore, we show that the similarity matrix structure correlates with answer accuracy, suggesting future directions for retrieval quality-aware document understanding.

Related papers

• R$^3$AG: Retriever Routing for Retrieval-Augmented Generation [59.47703698994575]
  R$3$AG decomposes retriever capability into two learnable dimensions: retrieval quality and generation utility.<n>R$3$AG consistently outperforms both the best individual retrievers and state-of-the-art static routing methods.
  arXiv  Detail & Related papers  (2026-04-22T06:51:20Z)
• Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA [71.42483000929614]
  Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents.<n>We propose Doc-$V*$, an textbfOCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation.
  arXiv  Detail & Related papers  (2026-04-15T11:12:27Z)
• NaviRAG: Towards Active Knowledge Navigation for Retrieval-Augmented Generation [50.16741209529908]
  NaviRAG is a novel framework that shifts from passive segment retrieval to active knowledge navigation.<n>We show that NaviRAG consistently improves both retrieval recall and end-to-end answer performance over conventional RAG baselines.
  arXiv  Detail & Related papers  (2026-04-14T14:07:01Z)
• LITTA: Late-Interaction and Test-Time Alignment for Visually-Grounded Multimodal Retrieval [0.0]
  LITTA is a query-expansion-centric retrieval framework for evidence page retrieval.<n>Given a user query, LITTA generates complementary query variants using a large language model.<n>Candidates from expanded queries are aggregated through reciprocal rank fusion to improve evidence coverage.
  arXiv  Detail & Related papers  (2026-03-10T13:25:39Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Query Decomposition for RAG: Balancing Exploration-Exploitation [83.79639293409802]
  RAG systems address complex user requests by decomposing them into subqueries, retrieving potentially relevant documents for each, and then aggregating them to generate an answer.<n>We formulate query decomposition and document retrieval in an exploitation-exploration setting, where retrieving one document at a time builds a belief about the utility of a given sub-queries.<n>Our main finding is that estimating document relevance using rank information and human judgments yields a 35% gain in document-level precision, 15% increase in alpha-nDCG, and better performance on the downstream task of long-form generation.
  arXiv  Detail & Related papers  (2025-10-21T13:37:11Z)
• LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding [37.12229829548839]
  We propose LAD-RAG, a novel layout-Aware Dynamic RAG framework.<n>LAD-RAG constructs a symbolic document graph that captures layout structure and cross-page dependencies.<n>Experiments on MMLongBench-Doc, LongDocURL, DUDE, and MP-DocVQA demonstrate that LAD-RAG improves retrieval, achieving over 90% perfect recall on average without any top-k tuning.
  arXiv  Detail & Related papers  (2025-10-08T17:02:04Z)
• Cluster-based Adaptive Retrieval: Dynamic Context Selection for RAG Applications [8.946586077722822]
  Cluster-based Adaptive Retrieval (CAR) is an algorithm that determines the optimal number of documents by analyzing the clustering patterns of ordered query-document similarity distances.<n>CAR consistently picks the optimal retrieval depth and achieves the highest TES score, outperforming every fixed top-k baseline.
  arXiv  Detail & Related papers  (2025-10-02T05:11:12Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• DR-RAG: Applying Dynamic Document Relevance to Retrieval-Augmented Generation for Question-Answering [4.364937306005719]
  RAG has recently demonstrated the performance of Large Language Models (LLMs) in the knowledge-intensive tasks such as Question-Answering (QA) We have found that even though there is low relevance between some critical documents and query, it is possible to retrieve the remaining documents by combining parts of the documents with the query. A two-stage retrieval framework called Dynamic-Relevant Retrieval-Augmented Generation (DR-RAG) is proposed to improve document retrieval recall and the accuracy of answers.
  arXiv  Detail & Related papers  (2024-06-11T15:15:33Z)
• Corrective Retrieval Augmented Generation [36.04062963574603]
  Retrieval-augmented generation (RAG) relies heavily on relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. We propose the Corrective Retrieval Augmented Generation (CRAG) to improve the robustness of generation. CRAG is plug-and-play and can be seamlessly coupled with various RAG-based approaches.
  arXiv  Detail & Related papers  (2024-01-29T04:36:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.