Transformer-Based UNet with Multi-Headed Cross-Attention Skip Connections to Eliminate Artifacts in Scanned Documents

• URL: http://arxiv.org/abs/2306.02815v1
• Date: Mon, 5 Jun 2023 12:12:23 GMT
• Title: Transformer-Based UNet with Multi-Headed Cross-Attention Skip Connections to Eliminate Artifacts in Scanned Documents
• Authors: David Kreuzer and Michael Munz
• Abstract summary: A modified UNet structure using a Swin Transformer backbone is presented to remove typical artifacts in scanned documents. An improvement in text extraction quality with a reduced error rate of up to 53.9% on the synthetic data is archived.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The extraction of text in high quality is essential for text-based document analysis tasks like Document Classification or Named Entity Recognition. Unfortunately, this is not always ensured, as poor scan quality and the resulting artifacts lead to errors in the Optical Character Recognition (OCR) process. Current approaches using Convolutional Neural Networks show promising results for background removal tasks but fail correcting artifacts like pixelation or compression errors. For general images, Transformer backbones are getting integrated more frequently in well-known neural network structures for denoising tasks. In this work, a modified UNet structure using a Swin Transformer backbone is presented to remove typical artifacts in scanned documents. Multi-headed cross-attention skip connections are used to more selectively learn features in respective levels of abstraction. The performance of this approach is examined regarding compression errors, pixelation and random noise. An improvement in text extraction quality with a reduced error rate of up to 53.9% on the synthetic data is archived. The pretrained base-model can be easily adapted to new artifacts. The cross-attention skip connections allow to integrate textual information extracted from the encoder or in form of commands to more selectively control the models outcome. The latter is shown by means of an example application.

Related papers

• Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation [67.89838237013078]
  Named entity recognition (NER) models often struggle with noisy inputs. We propose a more realistic setting in which only noisy text and its NER labels are available. We employ a multi-view training framework that improves robust NER without retrieving text during inference.
  arXiv  Detail & Related papers  (2024-07-26T07:30:41Z)
• MixTex: Unambiguous Recognition Should Not Rely Solely on Real Data [0.0]
  This paper introduces MixTex, an end-to-end OCR model designed for low-bias multilingual recognition. We identify specific recognition bias issues, such as the frequent misinterpretation of $e-t$ as $e-t$. We propose an innovative data augmentation method to mitigate this bias.
  arXiv  Detail & Related papers  (2024-06-24T21:38:36Z)
• DocDiff: Document Enhancement via Residual Diffusion Models [7.972081359533047]
  We propose DocDiff, a diffusion-based framework specifically designed for document enhancement problems. DocDiff consists of two modules: the Coarse Predictor (CP) and the High-Frequency Residual Refinement (HRR) module. Our proposed HRR module in pre-trained DocDiff is plug-and-play and ready-to-use, with only 4.17M parameters.
  arXiv  Detail & Related papers  (2023-05-06T01:41:10Z)
• ClipCrop: Conditioned Cropping Driven by Vision-Language Model [90.95403416150724]
  We take advantage of vision-language models as a foundation for creating robust and user-intentional cropping algorithms. We develop a method to perform cropping with a text or image query that reflects the user's intention as guidance. Our pipeline design allows the model to learn text-conditioned aesthetic cropping with a small dataset.
  arXiv  Detail & Related papers  (2022-11-21T14:27:07Z)
• Boosting Modern and Historical Handwritten Text Recognition with Deformable Convolutions [52.250269529057014]
  Handwritten Text Recognition (HTR) in free-volution pages is a challenging image understanding task. We propose to adopt deformable convolutions, which can deform depending on the input at hand and better adapt to the geometric variations of the text.
  arXiv  Detail & Related papers  (2022-08-17T06:55:54Z)
• Noise and Edge Based Dual Branch Image Manipulation Detection [9.400611271697302]
  In this paper, the noise image extracted by the improved constrained convolution is used as the input of the model. The dual-branch network, consisting of a high-resolution branch and a context branch, is used to capture the traces of artifacts as much as possible. A specially designed manipulation edge detection module is constructed based on the dual-branch network to identify these artifacts better.
  arXiv  Detail & Related papers  (2022-07-02T03:28:51Z)
• Unsupervised Structure-Texture Separation Network for Oracle Character Recognition [70.29024469395608]
  Oracle bone script is the earliest-known Chinese writing system of the Shang dynasty and is precious to archeology and philology. We propose a structure-texture separation network (STSN), which is an end-to-end learning framework for joint disentanglement, transformation, adaptation and recognition.
  arXiv  Detail & Related papers  (2022-05-13T10:27:02Z)
• DocScanner: Robust Document Image Rectification with Progressive Learning [162.03694280524084]
  This work presents DocScanner, a new deep network architecture for document image rectification. DocScanner maintains a single estimate of the rectified image, which is progressively corrected with a recurrent architecture. The iterative refinements make DocScanner converge to a robust and superior performance, and the lightweight recurrent architecture ensures the running efficiency.
  arXiv  Detail & Related papers  (2021-10-28T09:15:02Z)
• Lights, Camera, Action! A Framework to Improve NLP Accuracy over OCR documents [2.6201102730518606]
  We demonstrate an effective framework for mitigating OCR errors for any downstream NLP task. We first address the data scarcity problem for model training by constructing a document synthesis pipeline. For the benefit of the community, we have made the document synthesis pipeline available as an open-source project.
  arXiv  Detail & Related papers  (2021-08-06T00:32:54Z)
• Robust Document Representations using Latent Topics and Metadata [17.306088038339336]
  We propose a novel approach to fine-tuning a pre-trained neural language model for document classification problems. We generate document representations that capture both text and metadata artifacts in a task manner. Our solution also incorporates metadata explicitly rather than just augmenting them with text.
  arXiv  Detail & Related papers  (2020-10-23T21:52:38Z)
• End-to-End Object Detection with Transformers [88.06357745922716]
  We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss.
  arXiv  Detail & Related papers  (2020-05-26T17:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.