DLoRA-TrOCR: Mixed Text Mode Optical Character Recognition Based On Transformer

• URL: http://arxiv.org/abs/2404.12734v3
• Date: Tue, 23 Apr 2024 05:36:31 GMT
• Title: DLoRA-TrOCR: Mixed Text Mode Optical Character Recognition Based On Transformer
• Authors: Da Chang, Yu Li,
• Abstract summary: Multi- fonts, mixed scenes and complex layouts seriously affect the recognition accuracy of traditional OCR models. We propose a parameter-efficient mixed text recognition method based on pre-trained OCR Transformer, namely DLoRA-TrOCR.
• Score: 12.966765239586994
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the continuous development of Optical Character Recognition (OCR) and the expansion of application fields, text recognition in complex scenes has become a key challenge. Factors such as multiple fonts, mixed scenes and complex layouts seriously affect the recognition accuracy of traditional OCR models. Although OCR models based on deep learning have performed well in specific fields or similar datasets in recent years, the generalization ability and robustness of the model are still a big challenge when facing complex environments with multiple scenes. Furthermore, training an OCR model from scratch or fine-tuning all parameters is very demanding on computing resources and inference time, which limits the flexibility of its application. This study focuses on a fundamental aspect of mixed text recognition in response to the challenges mentioned above, which involves effectively fine-tuning the pre-trained basic OCR model to demonstrate exceptional performance across various downstream tasks. To this end, we propose a parameter-efficient mixed text recognition method based on pre-trained OCR Transformer, namely DLoRA-TrOCR. This method embeds DoRA into the image encoder and LoRA into the internal structure of the text decoder, enabling efficient parameter fine-tuning for downstream tasks. Experiments show that compared to similar parameter adjustment methods, our model DLoRA-TrOCR has the smallest number of parameters and performs better. It can achieve state-of-the-art performance on complex scene datasets involving simultaneous recognition of mixed handwritten, printed and street view texts.

Related papers

• VISTA-OCR: Towards generative and interactive end to end OCR models [3.7548609506798494]
  VISTA-OCR is a lightweight architecture that unifies text detection and recognition within a single generative model. Built on an encoder-decoder architecture, VISTA-OCR is progressively trained, starting with the visual feature extraction phase. To enhance the model's capabilities, we built a new dataset composed of real-world examples enriched with bounding box annotations and synthetic samples.
  arXiv  Detail & Related papers  (2025-04-04T17:39:53Z)
• See then Tell: Enhancing Key Information Extraction with Vision Grounding [54.061203106565706]
  We introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding. To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets.
  arXiv  Detail & Related papers  (2024-09-29T06:21:05Z)
• UNIT: Unifying Image and Text Recognition in One Vision Encoder [51.140564856352825]
  UNIT is a novel training framework aimed at UNifying Image and Text recognition within a single model. We show that UNIT significantly outperforms existing methods on document-related tasks. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment.
  arXiv  Detail & Related papers  (2024-09-06T08:02:43Z)
• Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation [67.89838237013078]
  Named entity recognition (NER) models often struggle with noisy inputs. We propose a more realistic setting in which only noisy text and its NER labels are available. We employ a multi-view training framework that improves robust NER without retrieving text during inference.
  arXiv  Detail & Related papers  (2024-07-26T07:30:41Z)
• Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering [8.382903851560595]
  Scene-Text Visual Question Answering (ST-VQA) aims to understand scene text in images and answer questions related to the text content. Most existing methods heavily rely on the accuracy of Optical Character Recognition (OCR) systems. We propose a multimodal adversarial training architecture with spatial awareness capabilities.
  arXiv  Detail & Related papers  (2024-03-14T11:22:06Z)
• SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation [39.84456803546365]
  SSR-Encoder is a novel architecture designed for selectively capturing any subject from single or multiple reference images. It responds to various query modalities including text and masks, without necessitating test-time fine-tuning. Characterized by its model generalizability and efficiency, the SSR-Encoder adapts to a range of custom models and control modules.
  arXiv  Detail & Related papers  (2023-12-26T14:39:11Z)
• UPOCR: Towards Unified Pixel-Level OCR Interface [36.966005829678124]
  We propose UPOCR, a simple-yet-effective generalist model for Unified Pixel-level OCR interface. Specifically, the UPOCR unifies the paradigm of diverse OCR tasks as image-to-image transformation and the architecture as a vision Transformer (ViT)-based encoder-decoder. Experiments are conducted on three pixel-level OCR tasks including text removal, text segmentation, and tampered text detection.
  arXiv  Detail & Related papers  (2023-12-05T11:53:17Z)
• Donut: Document Understanding Transformer without OCR [17.397447819420695]
  We propose a novel VDU model that is end-to-end trainable without underpinning OCR framework. Our approach achieves state-of-the-art performance on various document understanding tasks in public benchmark datasets and private industrial service datasets.
  arXiv  Detail & Related papers  (2021-11-30T18:55:19Z)
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models [47.48019831416665]
  We propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR. TrOCR is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on both printed and handwritten text recognition tasks.
  arXiv  Detail & Related papers  (2021-09-21T16:01:56Z)
• Rethinking Text Line Recognition Models [57.47147190119394]
  We consider two decoder families (Connectionist Temporal Classification and Transformer) and three encoder modules (Bidirectional LSTMs, Self-Attention, and GRCLs) We compare their accuracy and performance on widely used public datasets of scene and handwritten text. Unlike the more common Transformer-based models, this architecture can handle inputs of arbitrary length.
  arXiv  Detail & Related papers  (2021-04-15T21:43:13Z)
• Scene Text Image Super-Resolution in the Wild [112.90416737357141]
  Low-resolution text images are often seen in natural scenes such as documents captured by mobile phones. Previous single image super-resolution (SISR) methods are trained on synthetic low-resolution images. We pro-pose a real scene text SR dataset, termed TextZoom. It contains paired real low-resolution and high-resolution images captured by cameras with different focal length in the wild.
  arXiv  Detail & Related papers  (2020-05-07T09:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.