A Picture is Worth a Thousand Words? An Empirical Study of Aggregation Strategies for Visual Financial Document Retrieval

• URL: http://arxiv.org/abs/2605.14581v1
• Date: Thu, 14 May 2026 08:53:40 GMT
• Title: A Picture is Worth a Thousand Words? An Empirical Study of Aggregation Strategies for Visual Financial Document Retrieval
• Abstract summary: Visual RAG treats documents as images and uses vision encoders to obtain vision patch tokens.<n>Hundreds of patch tokens per document create retrieval and storage challenges in a vector database.<n>This raises a critical question: does single-vector aggregation lose key information in financial documents?
• Score: 8.560410767537162
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Visual RAG has offered an alternative to traditional RAG. It treats documents as images and uses vision encoders to obtain vision patch tokens. However, hundreds of patch tokens per document create retrieval and storage challenges in a vector database. Practical deployment requires aggregating them into a single vector. This raises a critical question: does single-vector aggregation lose key information in financial documents? We develop a diagnostic benchmark using financial documents where changes in single digits can lead to significant semantic shifts. Our experiments show that single-vector aggregation collapses different documents with almost identical vectors. Metrics show that the patch level detects semantic changes, and confirm that aggregation obscures these details. We identify global texture dominance as the root cause. Our findings are consistent across model scales, retrieval-optimized embeddings, and multiple mitigation strategies, highlighting significant risks for single-vector visual document retrieval in financial applications.

Related papers

• PDMR: Passage-Driven Multi-ID Document Retrieval [1.1661238776379117]
  Passage-Driven Multi-ID Retrieval (PDMR) is a generative retrieval framework that represents documents through multiple passage-level identifiers.<n>PDMR segments each document and assigns one identifier to each selected passage, which provides multiple semantic entry points for retrieving the same document.
  arXiv  Detail & Related papers  (2026-09-08T13:58:59Z)
• Lost in a Single Vector: Improving Long-Document Retrieval with Chunk Evidence Aggregation [59.438696079345426]
  DICE is a training-free document-side strategy that splits documents into chunks, encodes them independently with a frozen model, and aggregates them back into a single vector.<n>On LongEmbed, DICE improves retrieval across four backbones, with the largest gains on slices beyond 4k tokens.<n>Across 12,779 filtered samples, DICE yields lower EDI than the single-vector baseline in 92.8% of cases.
  arXiv  Detail & Related papers  (2026-06-17T07:44:04Z)
• Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation [19.889854990300595]
  Iterative Retrieval-Augmented Generation (iRAG) has emerged as a powerful paradigm for answering complex multi-hop questions.<n>We present textbfChain of Evidence (CoE), a retriever-agnostic visual attribution framework.
  arXiv  Detail & Related papers  (2026-05-02T06:40:42Z)
• Multi-Vector Index Compression in Any Modality [73.7330345057813]
  Late interaction has emerged as a dominant paradigm for information retrieval in text, images, visual documents, and videos.<n>We introduce four approaches for index compression: sequence resizing, memory tokens, hierarchical pooling, and a novel attention-guided clustering (AGC)<n>AGC uses an attention-guided mechanism to identify the most semantically salient regions of a document as cluster centroids and to weight token aggregation.
  arXiv  Detail & Related papers  (2026-02-24T18:57:33Z)
• Beyond Patch Aggregation: 3-Pass Pyramid Indexing for Vision-Enhanced Document Retrieval [0.0]
  Document centric RAG pipelines usually begin with OCR, followed by brittles for chunking, table parsing, and layout reconstruction.<n>We introduce VisionRAG, a multimodal retrieval system that is OCR free and model agnostic.<n> VisionRAG indexes documents directly as images, preserving layout, tables, and spatial cues, and builds semantic vectors without committing to a specific extraction.
  arXiv  Detail & Related papers  (2025-11-26T07:18:06Z)
• UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG [82.84014669683863]
  Multimodal retrieval-augmented generation (MM-RAG) is a key approach for applying large language models to real-world knowledge bases.<n>UniDoc-Bench is the first large-scale, realistic benchmark for MM-RAG built from 70k real-world PDF pages.<n>Our experiments show that multimodal text-image fusion RAG systems consistently outperform both unimodal and jointly multimodal embedding-based retrieval.
  arXiv  Detail & Related papers  (2025-10-04T04:30:13Z)
• Learning Refined Document Representations for Dense Retrieval via Deliberate Thinking [58.69615583599489]
  Deliberate Thinking based Retriever (Debater) is a novel approach that enhances document representations by incorporating a step-by-step thinking process.<n>Debater significantly outperforms existing methods across several retrieval benchmarks.
  arXiv  Detail & Related papers  (2025-02-18T15:56:34Z)
• SelfDocSeg: A Self-Supervised vision-based Approach towards Document Segmentation [15.953725529361874]
  Document layout analysis is a known problem to the documents research community. With growing internet connectivity to personal life, an enormous amount of documents had been available in the public domain. We address this challenge using self-supervision and unlike, the few existing self-supervised document segmentation approaches.
  arXiv  Detail & Related papers  (2023-05-01T12:47:55Z)
• Same or Different? Diff-Vectors for Authorship Analysis [78.83284164605473]
  In classic'' authorship analysis a feature vector represents a document, the value of a feature represents (an increasing function of) the relative frequency of the feature in the document, and the class label represents the author of the document. Our experiments tackle same-author verification, authorship verification, and closed-set authorship attribution; while DVs are naturally geared for solving the 1st, we also provide two novel methods for solving the 2nd and 3rd.
  arXiv  Detail & Related papers  (2023-01-24T08:48:12Z)
• Precise Zero-Shot Dense Retrieval without Relevance Labels [60.457378374671656]
  Hypothetical Document Embeddings(HyDE) is a zero-shot dense retrieval system. We show that HyDE significantly outperforms the state-of-the-art unsupervised dense retriever Contriever.
  arXiv  Detail & Related papers  (2022-12-20T18:09:52Z)
• Multi-Vector Retrieval as Sparse Alignment [21.892007741798853]
  We propose a novel multi-vector retrieval model that learns sparsified pairwise alignments between query and document tokens. We learn the sparse unary saliences with entropy-regularized linear programming, which outperforms other methods to achieve sparsity. Our model often produces interpretable alignments and significantly improves its performance when from larger language models.
  arXiv  Detail & Related papers  (2022-11-02T16:49:58Z)
• Multi-View Document Representation Learning for Open-Domain Dense Retrieval [87.11836738011007]
  This paper proposes a multi-view document representation learning framework. It aims to produce multi-view embeddings to represent documents and enforce them to align with different queries. Experiments show our method outperforms recent works and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2022-03-16T03:36:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.