Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

• URL: http://arxiv.org/abs/2607.07179v1
• Date: Wed, 08 Jul 2026 09:13:59 GMT
• Title: Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering
• Abstract summary: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge.<n>Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks.<n>We present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains.
• Score: 14.473567450006868
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Document Visual Question Answering (DocVQA) presents a complex multimodal challenge, requiring models to exploit visual, textual, and layout information from documents. Although Vision-Language Models (VLMs) have shown remarkable performance in text-vision tasks, their robustness and transferability to different document domains remains underexplored. In this study, we present a comprehensive evaluation of 8 open-source pretrained VLMs on DocVQA in three different document domains: industrial documents of varying type, infographics, and presentation slides. We systematically assess model performance under zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot learning evaluations of knowledge transfer between domains. Our findings demonstrate that while large pretrained VLMs possess strong zero-shot baselines for structured layouts, their performance strongly decreases on visually complex layouts of infographics and slides. Although parameter scaling is a dominant factor on performance, supervised finetuning yields higher relative gains in smaller architectures. Furthermore, our cross-domain and few-shot experiments show that visual understanding is the main bottleneck for DocVQA, not a lack of knowledge from the VLMs. Using 50 target domain samples, the models finetuned in DocVQA with datasets of different domains rapidly adapt to the target domain documents, even surpassing their fully supervised counterparts in some cases.

Related papers

• FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios [58.34124792457706]
  The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution.<n>Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets.<n>We first construct a high-quality dataset that combines realworld 2D images and 3D point clouds, annotated with fine-grained domain semantics.<n>We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely verification, structural surface inspection, and assembly verification, revealing significant performance gaps.
  arXiv  Detail & Related papers  (2026-04-08T12:23:27Z)
• How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing [56.60465182650588]
  We introduce three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning.<n>We propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment.<n>We find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models.
  arXiv  Detail & Related papers  (2026-02-02T09:24:45Z)
• Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models [5.310444614342132]
  multimodal vision-language models (VLMs) may offer a flexible alternative for open-set object counting.<n>VLMs can approximately enumerate the number of items in a visual scene, matching or even surpassing the performance of specialized computer vision architectures.<n>None of the models can reliably count the number of objects in complex visual scenes.
  arXiv  Detail & Related papers  (2025-12-17T09:56:25Z)
• Scaling Beyond Context: A Survey of Multimodal Retrieval-Augmented Generation for Document Understanding [61.36285696607487]
  Document understanding is critical for applications from financial analysis to scientific discovery.<n>Current approaches, whether OCR-based pipelines feeding Large Language Models (LLMs) or native Multimodal LLMs (MLLMs) face key limitations.<n>Retrieval-Augmented Generation (RAG) helps ground models in external data, but documents' multimodal nature, combining text, tables, charts, and layout, demands a more advanced paradigm: Multimodal RAG.
  arXiv  Detail & Related papers  (2025-10-17T02:33:16Z)
• RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering [50.42577862494645]
  We present RAG-IGBench, a benchmark designed to evaluate the task of Interleaved Generation based on Retrieval-Augmented Generation (RAG-IG) in open-domain question answering.<n>RAG-IG integrates multimodal large language models (MLLMs) with retrieval mechanisms, enabling the models to access external image-text information for generating coherent multimodal content.
  arXiv  Detail & Related papers  (2025-10-11T03:06:39Z)
• Evaluating the Robustness of Open-Source Vision-Language Models to Domain Shift in Object Captioning [4.180203626942459]
  Vision-Language Models (VLMs) have emerged as powerful tools for generating textual descriptions from visual data.<n>This paper presents a systematic evaluation of VLM performance on a single-view object captioning task.<n>We compare captioning accuracy across two distinct object sets: a collection of multi-material, real-world tools and a set of single-material, 3D-printed items.
  arXiv  Detail & Related papers  (2025-06-24T12:45:09Z)
• QID: Efficient Query-Informed ViTs in Data-Scarce Regimes for OCR-free Visual Document Understanding [53.69841526266547]
  Fine-tuning a pre-trained Vision-Language Model with new datasets often falls short in optimizing the vision encoder.<n>We introduce QID, a novel, streamlined, architecture-preserving approach that integrates query embeddings into the vision encoder.
  arXiv  Detail & Related papers  (2025-04-03T18:47:16Z)
• How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model? [2.3993515715868714]
  We propose a novel, generalizable methodology to identify preferred image distributions for Vision-Language Models (VLMs) Applying this to different rendering types of 3D objects, we demonstrate its efficacy across various domains requiring precise interpretation of complex structures. To address the lack of benchmarks in specialized domains, we introduce CAD-VQA, a new dataset for evaluatingVLMs on CAD-related visual question answering tasks.
  arXiv  Detail & Related papers  (2024-09-03T19:26:13Z)
• On Task-personalized Multimodal Few-shot Learning for Visually-rich Document Entity Retrieval [59.25292920967197]
  Few-shot document entity retrieval (VDER) is an important topic in industrial NLP applications. FewVEX is a new dataset to boost future research in the field of entity-level few-shot VDER. We present a task-aware meta-learning based framework, with a central focus on achieving effective task personalization.
  arXiv  Detail & Related papers  (2023-11-01T17:51:43Z)
• GVdoc: Graph-based Visual Document Classification [17.350393956461783]
  We propose GVdoc, a graph-based document classification model. Our approach generates a document graph based on its layout, and then trains a graph neural network to learn node and graph embeddings. We show that our model, even with fewer parameters, outperforms state-of-the-art models on out-of-distribution data.
  arXiv  Detail & Related papers  (2023-05-26T19:23:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.