ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval

• URL: http://arxiv.org/abs/2608.15698v2
• Date: Fri, 21 Aug 2026 15:34:51 GMT
• Title: ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval
• Abstract summary: ConceptFormer is a latent concept representation learning framework for visual document retrieval.<n>It query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance.<n>It achieves 16.7% and 22.1% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively.
• Score: 62.80735412748694
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.

Related papers

• Thinking with Anchors: Grounded and Efficient Document Reasoning [87.07530736788898]
  We present ADOPD 2026, a reasoning-oriented extension of ADOPD.<n>ADOPD 2026 turns page decomposition into spatially grounded document understanding.<n>By connecting page decomposition to verifiable visual-anchor reasoning, ADOPD 2026 provides a task framework.
  arXiv  Detail & Related papers  (2026-08-05T04:09:05Z)
• VaRS-Doc: Interpretation-Aware Variant Representations via Latent Self-Probing for Visual Document Retrieval [40.36520318905935]
  We propose VaRS-Doc, a visual document retrieval framework that diversifies document representations.<n>VaRS-Doc achieves state-of-the-art retrieval performance, offering a practical solution to the mismatch between query-agnostic document encoding and query-specific retrieval needs.
  arXiv  Detail & Related papers  (2026-08-02T12:55:57Z)
• MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing [74.84107522458798]
  MPDocBench-Parse is a benchmark for multi-page document parsing in real-world applications.<n>It contains 433 manually annotated documents with 3,246 pages, covering 15 document types in English and Chinese, with diverse layout styles.
  arXiv  Detail & Related papers  (2026-05-21T07:36:41Z)
• Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation [19.889854990300595]
  Iterative Retrieval-Augmented Generation (iRAG) has emerged as a powerful paradigm for answering complex multi-hop questions.<n>We present textbfChain of Evidence (CoE), a retriever-agnostic visual attribution framework.
  arXiv  Detail & Related papers  (2026-05-02T06:40:42Z)
• ReAlign: Optimizing the Visual Document Retriever with Reasoning-Guided Fine-Grained Alignment [28.897559367200376]
  Visual document retrieval aims to retrieve a set of document pages relevant to a query from visually rich collections.<n>Existing methods often employ Vision-Language Models (VLMs) to encode queries and visual pages into a shared embedding space.<n>We propose Reasoning-Guided Alignment (ReAlign) to provide fine-grained visual document descriptions as supervision signals for training.
  arXiv  Detail & Related papers  (2026-04-08T14:47:27Z)
• MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns [80.05126590825121]
  MonkeyOCR v1.5 is a unified vision-language framework that enhances both layout understanding and content recognition.<n>To address complex table structures, we propose a visual consistency-based reinforcement learning scheme.<n>Two specialized modules, Image-Decoupled Table Parsing and Type-Guided Table Merging, are introduced to enable reliable parsing of tables.
  arXiv  Detail & Related papers  (2025-11-13T15:12:17Z)
• Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering [51.7493726399073]
  We present a discourse-aware hierarchical framework to enhance long document question answering.<n>The framework involves three key innovations: specialized discourse parsing for lengthy documents, LLM-based enhancement of discourse relation nodes, and structure-guided hierarchical retrieval.
  arXiv  Detail & Related papers  (2025-05-26T14:45:12Z)
• Adaptive Markup Language Generation for Contextually-Grounded Visual Document Understanding [42.506971197471195]
  We introduce two fine-grained structured datasets: DocMark-Pile, comprising approximately 3.8M pretraining data pairs for document parsing, and DocMark-Instruct, featuring 624k fine-tuning data annotations for grounded instruction following.<n>Our proposed model significantly outperforms existing state-of-theart MLLMs across a range of visual document understanding benchmarks.
  arXiv  Detail & Related papers  (2025-05-08T17:37:36Z)
• QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding [53.69841526266547]
  Fine-tuning a pre-trained Vision-Language Model with new datasets often falls short in optimizing the vision encoder.<n>We introduce QID, a novel, streamlined, architecture-preserving approach that integrates query embeddings into the vision encoder.
  arXiv  Detail & Related papers  (2025-04-03T18:47:16Z)
• DocTr: Document Transformer for Structured Information Extraction in Documents [36.1145541816468]
  We present a new formulation for structured information extraction from visually rich documents. It aims to address the limitations of existing IOB tagging or graph-based formulations. We represent an entity as an anchor word and a bounding box, and represent entity linking as the association between anchor words.
  arXiv  Detail & Related papers  (2023-07-16T02:59:30Z)
• Unified Pretraining Framework for Document Understanding [52.224359498792836]
  We present UDoc, a new unified pretraining framework for document understanding. UDoc is designed to support most document understanding tasks, extending the Transformer to take multimodal embeddings as input. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses.
  arXiv  Detail & Related papers  (2022-04-22T21:47:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.