DoPTA: Improving Document Layout Analysis using Patch-Text Alignment

• URL: http://arxiv.org/abs/2412.12902v2
• Date: Sun, 09 Mar 2025 14:17:02 GMT
• Title: DoPTA: Improving Document Layout Analysis using Patch-Text Alignment
• Authors: Nikitha SR, Tarun Ram Menta, Mausoom Sarkar,
• Abstract summary: We present a novel image-text alignment technique specially designed for leveraging the textual information in document images to improve performance on visual tasks.<n>Our document encoder model DoPTA - trained with this technique demonstrates strong performance on a wide range of document image understanding tasks, without requiring OCR during inference.<n>DoPTA also sets new state-of-the art results on D4LA, and FUNSD, two challenging document visual analysis benchmarks.
• Score: 3.3181276611945267
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The advent of multimodal learning has brought a significant improvement in document AI. Documents are now treated as multimodal entities, incorporating both textual and visual information for downstream analysis. However, works in this space are often focused on the textual aspect, using the visual space as auxiliary information. While some works have explored pure vision based techniques for document image understanding, they require OCR identified text as input during inference, or do not align with text in their learning procedure. Therefore, we present a novel image-text alignment technique specially designed for leveraging the textual information in document images to improve performance on visual tasks. Our document encoder model DoPTA - trained with this technique demonstrates strong performance on a wide range of document image understanding tasks, without requiring OCR during inference. Combined with an auxiliary reconstruction objective, DoPTA consistently outperforms larger models, while using significantly lesser pre-training compute. DoPTA also sets new state-of-the art results on D4LA, and FUNSD, two challenging document visual analysis benchmarks.

Related papers

• Visual Text Processing: A Comprehensive Review and Unified Evaluation [99.57846940547171]
  We present a comprehensive, multi-perspective analysis of recent advancements in visual text processing. Our aim is to establish this work as a fundamental resource that fosters future exploration and innovation in the dynamic field of visual text processing.
  arXiv  Detail & Related papers  (2025-04-30T14:19:29Z)
• InstructOCR: Instruction Boosting Scene Text Spotting [10.724187109801251]
  InstructOCR is an innovative instruction-based scene text spotting model. Our framework employs both text and image encoders during training and inference. We achieve state-of-the-art results on widely used benchmarks.
  arXiv  Detail & Related papers  (2024-12-20T03:23:26Z)
• LayoutLLM: Large Language Model Instruction Tuning for Visually Rich Document Understanding [0.0]
  This paper proposes LayoutLLM, a more flexible document analysis method for understanding imaged documents. Existing methods have been developed to enhance document comprehension by incorporating pre-training awareness of images, text, and layout structure. Our experiments demonstrate improvement over the baseline model in various document analysis tasks.
  arXiv  Detail & Related papers  (2024-03-21T09:25:24Z)
• Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs [96.54224331778195]
  We present a text-grounding document understanding model, termed TGDoc, which enhances MLLMs with the ability to discern the spatial positioning of text within images. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-11-22T06:46:37Z)
• DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding [91.17151775296234]
  This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding. Unlike existing work either struggle with high-resolution documents or give up the large language model thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space.
  arXiv  Detail & Related papers  (2023-11-20T14:42:25Z)
• Efficient End-to-End Visual Document Understanding with Rationale Distillation [43.28272448274713]
  Rationale Distillation (RD) trains a small student model to predict both rationales and answers. RD outperforms the base model by 4-5% absolute accuracy with only 1% higher computational cost.
  arXiv  Detail & Related papers  (2023-11-16T06:50:26Z)
• Unifying Vision, Text, and Layout for Universal Document Processing [105.36490575974028]
  We propose a Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites.
  arXiv  Detail & Related papers  (2022-12-05T22:14:49Z)
• Unified Pretraining Framework for Document Understanding [52.224359498792836]
  We present UDoc, a new unified pretraining framework for document understanding. UDoc is designed to support most document understanding tasks, extending the Transformer to take multimodal embeddings as input. An important feature of UDoc is that it learns a generic representation by making use of three self-supervised losses.
  arXiv  Detail & Related papers  (2022-04-22T21:47:04Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.