Scene Text Image Super-resolution based on Text-conditional Diffusion Models

• URL: http://arxiv.org/abs/2311.09759v2
• Date: Fri, 22 Dec 2023 09:30:39 GMT
• Title: Scene Text Image Super-resolution based on Text-conditional Diffusion Models
• Authors: Chihiro Noguchi, Shun Fukuda, Masao Yamanaka
• Abstract summary: Scene Text Image Super-resolution (STISR) has recently achieved great success as a preprocessing method for scene text recognition. In this study, we leverage text-conditional diffusion models (DMs) for STISR tasks. We propose a novel framework for LR-HR paired text image datasets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene Text Image Super-resolution (STISR) has recently achieved great success as a preprocessing method for scene text recognition. STISR aims to transform blurred and noisy low-resolution (LR) text images in real-world settings into clear high-resolution (HR) text images suitable for scene text recognition. In this study, we leverage text-conditional diffusion models (DMs), known for their impressive text-to-image synthesis capabilities, for STISR tasks. Our experimental results revealed that text-conditional DMs notably surpass existing STISR methods. Especially when texts from LR text images are given as input, the text-conditional DMs are able to produce superior quality super-resolution text images. Utilizing this capability, we propose a novel framework for synthesizing LR-HR paired text image datasets. This framework consists of three specialized text-conditional DMs, each dedicated to text image synthesis, super-resolution, and image degradation. These three modules are vital for synthesizing distinct LR and HR paired images, which are more suitable for training STISR methods. Our experiments confirmed that these synthesized image pairs significantly enhance the performance of STISR methods in the TextZoom evaluation.

Related papers

• Decoder Pre-Training with only Text for Scene Text Recognition [54.93037783663204]
  Scene text recognition (STR) pre-training methods have achieved remarkable progress, primarily relying on synthetic datasets. We introduce a novel method named Decoder Pre-training with only text for STR (DPTR) DPTR treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder.
  arXiv  Detail & Related papers  (2024-08-11T06:36:42Z)
• Visual Text Generation in the Wild [67.37458807253064]
  We propose a visual text generator (termed SceneVTG) which can produce high-quality text images in the wild. The proposed SceneVTG significantly outperforms traditional rendering-based methods and recent diffusion-based methods in terms of fidelity and reasonability. The generated images provide superior utility for tasks involving text detection and text recognition.
  arXiv  Detail & Related papers  (2024-07-19T09:08:20Z)
• ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting [8.397246652127793]
  We propose a new pre-training method called OCR-Text Destylization Modeling (ODM) ODM transfers diverse styles of text found in images to a uniform style based on the text prompt. Our method significantly improves performance and outperforms current pre-training methods in scene text detection and spotting tasks.
  arXiv  Detail & Related papers  (2024-03-01T06:13:53Z)
• Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models [63.99110667987318]
  We present DiffText, a pipeline that seamlessly blends foreground text with the background's intrinsic features. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors.
  arXiv  Detail & Related papers  (2023-11-28T06:51:28Z)
• Image Super-Resolution with Text Prompt Diffusion [118.023531454099]
  We introduce text prompts to image SR to provide degradation priors. PromptSR utilizes the pre-trained language model (e.g., T5 or CLIP) to enhance restoration. Experiments indicate that introducing text prompts into SR, yields excellent results on both synthetic and real-world images.
  arXiv  Detail & Related papers  (2023-11-24T05:11:35Z)
• TextDiff: Mask-Guided Residual Diffusion Models for Scene Text Image Super-Resolution [18.73348268987249]
  TextDiff is a diffusion-based framework tailored for scene text image super-resolution. It achieves state-of-the-art (SOTA) performance on public benchmark datasets. Our proposed MRD module is plug-and-play that effectively sharpens the text edges produced by SOTA methods.
  arXiv  Detail & Related papers  (2023-08-13T11:02:16Z)
• Text Prior Guided Scene Text Image Super-resolution [11.396781380648756]
  Scene text image super-resolution (STISR) aims to improve the resolution and visual quality of low-resolution (LR) scene text images. We make an attempt to embed categorical text prior into STISR model training. We present a multi-stage text prior guided super-resolution framework for STISR.
  arXiv  Detail & Related papers  (2021-06-29T12:52:33Z)
• DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis [80.54273334640285]
  We propose a novel one-stage text-to-image backbone that directly synthesizes high-resolution images without entanglements between different generators. We also propose a novel Target-Aware Discriminator composed of Matching-Aware Gradient Penalty and One-Way Output. Compared with current state-of-the-art methods, our proposed DF-GAN is simpler but more efficient to synthesize realistic and text-matching images.
  arXiv  Detail & Related papers  (2020-08-13T12:51:17Z)
• Scene Text Image Super-Resolution in the Wild [112.90416737357141]
  Low-resolution text images are often seen in natural scenes such as documents captured by mobile phones. Previous single image super-resolution (SISR) methods are trained on synthetic low-resolution images. We pro-pose a real scene text SR dataset, termed TextZoom. It contains paired real low-resolution and high-resolution images captured by cameras with different focal length in the wild.
  arXiv  Detail & Related papers  (2020-05-07T09:18:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.