VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization

• URL: http://arxiv.org/abs/2507.09531v1
• Date: Sun, 13 Jul 2025 08:15:11 GMT
• Title: VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization
• Authors: Son Nguyen, Giang Nguyen, Hung Dao, Thao Do, Daeyoung Kim,
• Abstract summary: Key Information Extraction underpins the understanding of visual documents.<n>Existing multimodal large language models (MLLMs) often perform poorly on dense documents.<n>We introduce VDInstruct, an MLLM that separates spatial region detection from semantic feature extraction.
• Score: 7.769156392417315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Key Information Extraction (KIE) underpins the understanding of visual documents (e.g., receipts and contracts) by extracting precise semantic content and accurately capturing spatial structure. Yet existing multimodal large language models (MLLMs) often perform poorly on dense documents and rely on vision tokenization approaches that scale with image size, leading to redundant computation and memory inefficiency. To address these challenges, we introduce VDInstruct, an MLLM that separates spatial region detection from semantic feature extraction. Central to our model is a content-aware tokenization strategy: rather than fragmenting the entire image uniformly, it generates tokens in proportion to document complexity, preserving critical structure while eliminating wasted tokens. Leveraging a three-stage training paradigm, our model achieves state-of-the-art (SOTA) results on KIE benchmarks, matching or exceeding the accuracy of leading approaches while reducing the number of image tokens by roughly 3.6x. In zero-shot evaluations, VDInstruct surpasses strong baselines-such as DocOwl 1.5-by +5.5 F1 points, highlighting its robustness to unseen documents. These findings show that content-aware tokenization combined with explicit layout modeling offers a promising direction forward for document understanding. Data, source code, and model weights will be made publicly available.

Related papers

• QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding [53.69841526266547]
  Fine-tuning a pre-trained Vision-Language Model with new datasets often falls short in optimizing the vision encoder.<n>We introduce QID, a novel, streamlined, architecture-preserving approach that integrates query embeddings into the vision encoder.
  arXiv  Detail & Related papers  (2025-04-03T18:47:16Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• Éclair -- Extracting Content and Layout with Integrated Reading Order for Documents [7.358946120326249]
  We introduce 'Eclair, a text-extraction tool specifically designed to process a wide range of document types.<n>Given an image, 'Eclair is able to extract formatted text in reading order, along with bounding boxes and their corresponding semantic classes.<n>'Eclair achieves state-of-the-art accuracy on this benchmark, outperforming other methods across key metrics.
  arXiv  Detail & Related papers  (2025-02-06T17:07:22Z)
• HAND: Hierarchical Attention Network for Multi-Scale Handwritten Document Recognition and Layout Analysis [21.25786478579275]
  Handwritten document recognition is one of the most challenging tasks in computer vision.<n>Traditionally, this problem has been approached as two separate tasks, handwritten text recognition and layout analysis.<n>This paper introduces HAND, a novel end-to-end and segmentation-free architecture for simultaneous text recognition and layout analysis tasks.
  arXiv  Detail & Related papers  (2024-12-25T20:36:29Z)
• mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding [103.05835688963947]
  We propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%. Compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens.
  arXiv  Detail & Related papers  (2024-09-05T11:09:00Z)
• Matryoshka Multimodal Models [92.41824727506751]
  We propose M3: Matryoshka Multimodal Models, which learns to represent visual content as nested sets of visual tokens. We find that COCO-style benchmarks only need around 9 visual tokens to obtain accuracy similar to that of using all 576 tokens.
  arXiv  Detail & Related papers  (2024-05-27T17:59:56Z)
• Subobject-level Image Tokenization [60.80949852899857]
  Patch-based image tokenization ignores the morphology of the visual world.<n>Inspired by subword tokenization, we introduce subobject-level adaptive token segmentation.<n>We show that subobject tokenization enables faster convergence and better generalization while using fewer visual tokens.
  arXiv  Detail & Related papers  (2024-02-22T06:47:44Z)
• Efficient End-to-End Visual Document Understanding with Rationale Distillation [43.28272448274713]
  Rationale Distillation (RD) trains a small student model to predict both rationales and answers. RD outperforms the base model by 4-5% absolute accuracy with only 1% higher computational cost.
  arXiv  Detail & Related papers  (2023-11-16T06:50:26Z)
• Attention Where It Matters: Rethinking Visual Document Understanding with Selective Region Concentration [26.408343160223517]
  We propose a novel end-to-end document understanding model called SeRum. SeRum converts image understanding and recognition tasks into a local decoding process of the visual tokens of interest. We show that SeRum achieves state-of-the-art performance on document understanding tasks and competitive results on text spotting tasks.
  arXiv  Detail & Related papers  (2023-09-03T10:14:34Z)
• One-shot Key Information Extraction from Document with Deep Partial Graph Matching [60.48651298832829]
  Key Information Extraction (KIE) from documents improves efficiency, productivity, and security in many industrial scenarios. Existing supervised learning methods for the KIE task need to feed a large number of labeled samples and learn separate models for different types of documents. We propose a deep end-to-end trainable network for one-shot KIE using partial graph matching.
  arXiv  Detail & Related papers  (2021-09-26T07:45:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.