An LMM for Precisely Grounding Elements in Documents

• URL: http://arxiv.org/abs/2606.24118v1
• Date: Tue, 23 Jun 2026 04:09:05 GMT
• Title: An LMM for Precisely Grounding Elements in Documents
• Abstract summary: We introduce PreciseDoc, an LMM specifically designed for precise element grounding and can be further optimized for Document VQA tasks.<n>We construct challenging training data by two pipelines capable of mass-producing high-quality documents with paired metadata of fine-grained coordinates.<n>The model develops more real-world functions beyond straightforward localization of single text, such as locating personal information from CVs.
• Score: 44.35200586537557
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual grounding in documents is a crucial ability for Large Multimodal Models (LMMs) in areas such as document understanding, deep research and document error detection. However, existing approaches exhibit poor grounding precision in text-rich document images, often failing to accurately locate the critical document elements needed for reliable reasoning. To address this gap, we introduce PreciseDoc, an LMM specifically designed for precise element grounding and can be further optimized for Document VQA tasks. Specifically, to enhance the basic localization capability, we construct challenging training data by two pipelines capable of mass-producing high-quality documents with paired metadata of fine-grained coordinates, including synthetic hand-filled documents with camera effects. The model develops more real-world functions beyond straightforward localization of single text, such as locating personal information from CVs. Furthermore, we introduce a training paradigm for visual grounded reasoning where the grounding and reasoning are supervised jointly with reinforcement learning to improve the contribution of the grounded evidence. A comprehensive evaluation on various benchmarks demonstrates the advantage of the proposed data and methods in document spatial grounding and document understanding.

Related papers

• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding [63.257540233507626]
  We propose a paradigm that requires the model to execute a structured Analysis, Localization and Reasoning workflow.<n>We show it robustly generalizes from short-page training to ultra-long documents and is naturally synergistic with visual Retrieval-Augmented Generation systems.
  arXiv  Detail & Related papers  (2026-04-14T14:39:26Z)
• DISCO: Document Intelligence Suite for COmparative Evaluation [1.4425299138308667]
  Document intelligence requires accurate text extraction and reliable reasoning over document content.<n>We evaluate optical character recognition (OCR) pipelines and vision-language models (VLMs) separately on parsing and question answering across diverse document types.
  arXiv  Detail & Related papers  (2026-03-04T14:47:34Z)
• DOGR: Towards Versatile Visual Document Grounding and Referring [47.66205811791444]
  Grounding and referring capabilities have gained increasing attention for achieving detailed understanding and flexible user interaction.<n>We propose the DOcument Grounding and Referring data engine (DOGR-Engine), which generates two types of high-quality fine-grained document data.<n>Using the DOGR-Engine, we construct DOGR-Bench, a benchmark covering seven grounding and referring tasks across three document types.
  arXiv  Detail & Related papers  (2024-11-26T05:38:34Z)
• Layout-Aware Information Extraction for Document-Grounded Dialogue: Dataset, Method and Demonstration [75.47708732473586]
  We propose a layout-aware document-level Information Extraction dataset, LIE, to facilitate the study of extracting both structural and semantic knowledge from visually rich documents. LIE contains 62k annotations of three extraction tasks from 4,061 pages in product and official documents. Empirical results show that layout is critical for VRD-based extraction, and system demonstration also verifies that the extracted knowledge can help locate the answers that users care about.
  arXiv  Detail & Related papers  (2022-07-14T07:59:45Z)
• Unified Pretraining Framework for Document Understanding [52.224359498792836]
  We present UDoc, a new unified pretraining framework for document understanding. UDoc is designed to support most document understanding tasks, extending the Transformer to take multimodal embeddings as input. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses.
  arXiv  Detail & Related papers  (2022-04-22T21:47:04Z)
• Augmenting Document Representations for Dense Retrieval with Interpolation and Perturbation [49.940525611640346]
  Document Augmentation for dense Retrieval (DAR) framework augments the representations of documents with their Dense Augmentation and perturbations. We validate the performance of DAR on retrieval tasks with two benchmark datasets, showing that the proposed DAR significantly outperforms relevant baselines on the dense retrieval of both the labeled and unlabeled documents.
  arXiv  Detail & Related papers  (2022-03-15T09:07:38Z)
• The Law of Large Documents: Understanding the Structure of Legal Contracts Using Visual Cues [0.7425558351422133]
  We measure the impact of incorporating visual cues, obtained via computer vision methods, on the accuracy of document understanding tasks. Our method of segmenting documents based on structural metadata out-performs existing methods on four long-document understanding tasks.
  arXiv  Detail & Related papers  (2021-07-16T21:21:50Z)
• Evaluation of a Region Proposal Architecture for Multi-task Document Layout Analysis [0.685316573653194]
  Mask-RCNN architecture is designed to address the problem of baseline detection and region segmentation. We present experimental results on two handwritten text datasets and one handwritten music dataset. The analyzed architecture yields promising results, outperforming state-of-the-art techniques in all three datasets.
  arXiv  Detail & Related papers  (2021-06-22T14:07:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.