DocVLM: Make Your VLM an Efficient Reader

• URL: http://arxiv.org/abs/2412.08746v1
• Date: Wed, 11 Dec 2024 19:35:06 GMT
• Title: DocVLM: Make Your VLM an Efficient Reader
• Abstract summary: We introduce DocVLM, a method that integrates an OCR-based modality into Vision-Language Models (VLMs)<n>Our approach employs an OCR encoder to capture textual content and layout, compressing these into a compact set of learned queries incorporated into the VLM.<n>In limited-token regimes (448$times$448), DocVLM with 64 learned queries improves DocVQA results from 56.4% to 86.6% when integrated with InternVL2 and from 84.4% to 91.2% with Qwen2-VL.
• Score: 16.11678886224833
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) excel in diverse visual tasks but face challenges in document understanding, which requires fine-grained text processing. While typical visual tasks perform well with low-resolution inputs, reading-intensive applications demand high-resolution, resulting in significant computational overhead. Using OCR-extracted text in VLM prompts partially addresses this issue but underperforms compared to full-resolution counterpart, as it lacks the complete visual context needed for optimal performance. We introduce DocVLM, a method that integrates an OCR-based modality into VLMs to enhance document processing while preserving original weights. Our approach employs an OCR encoder to capture textual content and layout, compressing these into a compact set of learned queries incorporated into the VLM. Comprehensive evaluations across leading VLMs show that DocVLM significantly reduces reliance on high-resolution images for document understanding. In limited-token regimes (448$\times$448), DocVLM with 64 learned queries improves DocVQA results from 56.0% to 86.6% when integrated with InternVL2 and from 84.4% to 91.2% with Qwen2-VL. In LLaVA-OneVision, DocVLM achieves improved results while using 80% less image tokens. The reduced token usage allows processing multiple pages effectively, showing impressive zero-shot results on DUDE and state-of-the-art performance on MP-DocVQA, highlighting DocVLM's potential for applications requiring high-performance and efficiency.

Related papers

• LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement [40.36520318905935]
  LightSTAR is an efficient framework that decomposes visual document retrieval into keywords and semantically rich words.<n>LightSTAR achieves state-of-the-art retrieval accuracy while reducing end-to-end latency by several-fold.
  arXiv  Detail & Related papers  (2026-06-22T16:16:24Z)
• Index Light, Reason Deep: Deferred Visual Ingestion for Visual-Dense Document Question Answering [3.0028612998967894]
  This paper proposes a demand-side ingestion strategy for multimodal document question answering.<n>It achieves comparable overall accuracy at zero ingestion VLM cost.<n>It also supports interactive refinement and progressive caching, transforming the "QA accuracy" problem into a "page localization" problem.
  arXiv  Detail & Related papers  (2026-02-15T14:23:50Z)
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [95.89543460132413]
  Vision-language models (VLMs) have improved performance by increasing the number of visual tokens.<n>However, most real-world scenarios do not require such an extensive number of visual tokens.<n>We present a new paradigm for visual token compression, namely, VisionThink.
  arXiv  Detail & Related papers  (2025-07-17T17:59:55Z)
• MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios [66.59827827146262]
  We introduce the MME-VideoOCR benchmark, which encompasses a comprehensive range of video OCR application scenarios.<n>The benchmark consists of 1,464 videos with varying resolutions, aspect ratios, and durations, along with 2,000 meticulously curated, manually annotated question-answer pairs.<n>We evaluate 18 state-of-the-art MLLMs on MME-VideoOCR, revealing that even the best-performing model (Gemini-2.5 Pro) achieves an accuracy of only 73.7%.
  arXiv  Detail & Related papers  (2025-05-27T15:27:46Z)
• MLLM-Guided VLM Fine-Tuning with Joint Inference for Zero-Shot Composed Image Retrieval [50.062817677022586]
  Zero-Shot Image Retrieval (ZS-CIR) methods typically train adapters that convert reference images into pseudo-text tokens.<n>We propose MLLM-Guided VLM Fine-Tuning with Joint Inference (MVFT-JI) to construct two complementary training tasks using only unlabeled images.
  arXiv  Detail & Related papers  (2025-05-26T08:56:59Z)
• LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding [29.719450799231705]
  Vision-Language Models (VLMs) obtain frame-level understanding capabilities through multi-frame input. Video Large Language Models (Video-LLMs) capture temporal relationships within visual features but are limited by the scarcity of high-quality video-text datasets. We propose Lightweight Video Compression (LVC), a novel method featuring the Query-Attention Video Compression mechanism.
  arXiv  Detail & Related papers  (2025-04-09T12:51:10Z)
• M-DocSum: Do LVLMs Genuinely Comprehend Interleaved Image-Text in Document Summarization? [49.53982792497275]
  We investigate whether Large Vision-Language Models (LVLMs) genuinely comprehend interleaved image-text in the document. Existing document understanding benchmarks often assess LVLMs using question-answer formats. We introduce a novel and challenging Multimodal Document Summarization Benchmark (M-DocSum-Bench) M-DocSum-Bench comprises 500 high-quality arXiv papers, along with interleaved multimodal summaries aligned with human preferences.
  arXiv  Detail & Related papers  (2025-03-27T07:28:32Z)
• Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection [53.558449071113245]
  Vision-Language Models (VLMs) leverage aligned visual encoders to transform images into visual tokens, allowing them to be processed similarly to text by the backbone large language model (LLM) Recent advancements in vision-language modeling introduce image cropping techniques that feed all encoded sub-images into the model. We propose a lightweight, universal framework that seamlessly integrates with existing VLMs to enhance their ability to process finegrained details.
  arXiv  Detail & Related papers  (2025-03-14T18:33:31Z)
• AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding [63.09928907734156]
  AlignVLM is a vision-text alignment method that maps visual features to a weighted average of text embeddings. Our experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods.
  arXiv  Detail & Related papers  (2025-02-03T13:34:51Z)
• Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage [50.84150600032693]
  Multimodal large language models (MLLMs) excel at generating highly detailed captions but often produce hallucinations.<n>We propose a multiagent approach that leverages LLM-MLLM collaboration to correct given captions.<n>Our proposed method significantly enhances the factual accuracy of captions, even improving those generated by GPT-4V.
  arXiv  Detail & Related papers  (2024-12-20T01:37:22Z)
• Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation [109.5893580175657]
  In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision.<n>This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data.<n>We propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's hidden representations.
  arXiv  Detail & Related papers  (2024-12-12T18:55:18Z)
• Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering [44.54319663913782]
  We propose Retrieval-Augmented MLLM with Compressed Contexts (RACC) RACC achieves a state-of-the-art (SOTA) performance of 62.9% on OK-VQA. It significantly reduces inference latency by 22.0%-59.7% compared to the prominent RAVQA-v2.
  arXiv  Detail & Related papers  (2024-09-11T15:11:39Z)
• mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding [103.05835688963947]
  We propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%. Compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens.
  arXiv  Detail & Related papers  (2024-09-05T11:09:00Z)
• A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding [30.754200683466788]
  We introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM) for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues. It also shows enhanced performance in Key Information Extraction (KIE) and Visual Question Answering (VQA)
  arXiv  Detail & Related papers  (2024-07-02T06:29:05Z)
• DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming [33.40963475653868]
  DocKylin is a document-centric MLLM that performs visual content slimming at both the pixel and token levels.<n>We introduce an Adaptive Pixel Slimming (APS) preprocessing module to perform pixel-level slimming.<n>We also propose a novel Dynamic Token Slimming (DTS) module to conduct token-level slimming.
  arXiv  Detail & Related papers  (2024-06-27T11:28:36Z)
• HRVDA: High-Resolution Visual Document Assistant [32.51417315241559]
  We propose a High-Resolution Visual Document Assistant (HRVDA) to bridge the gap between MLLMs and visual document understanding. HRVDA employs a content filtering mechanism and an instruction filtering module to filter out the content-agnostic visual tokens and instruction-agnostic visual tokens. Our model achieves state-of-the-art performance across multiple document understanding datasets.
  arXiv  Detail & Related papers  (2024-04-10T11:10:50Z)
• Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models [56.76307866160105]
  We propose a contrastive learning framework, termed Document Object COntrastive learning (DoCo) DoCo leverages an auxiliary multimodal encoder to obtain the features of document objects and align them to the visual features generated by the vision encoder of Large Visual-Language Models (LVLMs) We demonstrate that the proposed DoCo serves as a plug-and-play pre-training method, which can be employed in the pre-training of various LVLMs without inducing any increase in computational complexity during the inference process.
  arXiv  Detail & Related papers  (2024-02-29T10:17:27Z)
• Language Models as Black-Box Optimizers for Vision-Language Models [62.80817942316398]
  Vision-language models (VLMs) pre-trained on web-scale datasets have demonstrated remarkable capabilities on downstream tasks when fine-tuned with minimal data. We aim to develop a black-box approach to optimize VLMs through natural language prompts.
  arXiv  Detail & Related papers  (2023-09-12T04:03:41Z)
• BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions [41.825273034537204]
  Vision Language Models (VLMs) cannot accurately interpret images infused with text. The present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. Our model significantly enhances performance in processing text-rich VQA benchmarks and in undertaking general (not particularly text-rich) VQA benchmarks.
  arXiv  Detail & Related papers  (2023-08-19T07:53:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.