MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

• URL: http://arxiv.org/abs/2607.11562v1
• Date: Mon, 13 Jul 2026 13:43:39 GMT
• Title: MonkeyOCRv2: A Visual-Text Foundation Model for Document AI
• Abstract summary: MonkeyOCRv2 is a visual-text pretrained model for document AI.<n> MonkeyDoc v2 is the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages.
• Score: 71.06391669976786
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Mainstream visual encoders are pretrained on natural images and cannot be effectively applied to document images without document-oriented adaptation, as dense text and fine-grained character strokes demand character-level visual perception. We present MonkeyOCRv2, a visual-text pretrained model for document AI. First, we construct MonkeyDoc v2, to our knowledge the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages. Second, we propose a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details. Extensive experiments are conducted on five representative document analysis tasks, including text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation. Replacing the original encoders with MonkeyOCRv2 consistently improves performance across all five tasks. Finally, we validate its effectiveness as the vision encoder of multimodal large language models on the more challenging tasks of document parsing and document understanding. Kept frozen and paired with a lightweight language model, it yields a 0.7B document parsing model that sets a new open-source state-of-the-art on MDPBench, a recent benchmark spanning digital-born and photographed documents across 17 languages, surpassing the previous best 3B dots.mocr by 2.8% absolute with a vision encoder roughly 11$\times$ smaller. The frozen encoder also powers a document understanding model that outperforms counterparts built on CLIP, DINO, and SAM across eight benchmarks under identical training settings. These results suggest that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right.

Related papers

• Document-as-Image Representations Fall Short for Scientific Retrieval [54.7382379182364]
  We introduce ArXivDoc, a new benchmark constructed from the underlying sources of scientific papers.<n>We show that document-as-image representations are consistently suboptimal, especially as document length increases.<n>Text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context.
  arXiv  Detail & Related papers  (2026-04-20T17:00:17Z)
• Multimodal OCR: Parse Anything from Documents [72.69545534962234]
  dots.mocr treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets.<n>It reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction.<n>It supports end-to-end training over heterogeneous document elements.
  arXiv  Detail & Related papers  (2026-03-13T14:42:21Z)
• SERVAL: Surprisingly Effective Zero-Shot Visual Document Retrieval Powered by Large Vision and Language Models [17.85605201420847]
  Visual Document Retrieval (VDR) typically operates as text-to-image retrieval using specialized bi-encoders trained to directly embed document images.<n>We revisit a zero-shot generate-and-encode pipeline: a vision-language model first produces a detailed textual description of each document image.<n>On the ViDoRe-v2 benchmark, the method reaches 63.4% nDCG@5, surpassing the strongest specialised multi-vector visual document encoder.
  arXiv  Detail & Related papers  (2025-09-18T21:11:13Z)
• DoPTA: Improving Document Layout Analysis using Patch-Text Alignment [3.3181276611945267]
  We present a novel image-text alignment technique specially designed for leveraging the textual information in document images to improve performance on visual tasks.<n>Our document encoder model DoPTA - trained with this technique demonstrates strong performance on a wide range of document image understanding tasks, without requiring OCR during inference.<n>DoPTA also sets new state-of-the art results on D4LA, and FUNSD, two challenging document visual analysis benchmarks.
  arXiv  Detail & Related papers  (2024-12-17T13:26:31Z)
• Unifying Multimodal Retrieval via Document Screenshot Embedding [92.03571344075607]
  Document Screenshot Embedding (DSE) is a novel retrieval paradigm that regards document screenshots as a unified input format.<n>We first craft the dataset of Wiki-SS, a 1.3M Wikipedia web page screenshots as the corpus to answer the questions from the Natural Questions dataset.<n>For example, DSE outperforms BM25 by 17 points in top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide retrieval, DSE significantly outperforms OCR text retrieval methods by over 15 points in nDCG@10.
  arXiv  Detail & Related papers  (2024-06-17T06:27:35Z)
• DocumentCLIP: Linking Figures and Main Body Text in Reflowed Documents [18.080447065002392]
  We propose DocumentCLIP to enforce vision-language pretraining models to comprehend the interaction between images and longer text within documents. Our model is beneficial for the real-world multimodal document understanding like news article, magazines, product descriptions, which contain linguistically and visually richer content.
  arXiv  Detail & Related papers  (2023-06-09T23:51:11Z)
• Unifying Vision, Text, and Layout for Universal Document Processing [105.36490575974028]
  We propose a Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites.
  arXiv  Detail & Related papers  (2022-12-05T22:14:49Z)
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking [83.09001231165985]
  We propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks.
  arXiv  Detail & Related papers  (2022-04-18T16:19:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.