WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models

• URL: http://arxiv.org/abs/2303.16576v2
• Date: Wed, 17 May 2023 09:20:09 GMT
• Title: WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models
• Authors: Konstantina Nikolaidou, George Retsinas, Vincent Christlein, Mathias Seuret, Giorgos Sfikas, Elisa Barney Smith, Hamam Mokayed, Marcus Liwicki
• Abstract summary: We present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data.
• Score: 8.334487584550185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-Image synthesis is the task of generating an image according to a specific text description. Generative Adversarial Networks have been considered the standard method for image synthesis virtually since their introduction. Denoising Diffusion Probabilistic Models are recently setting a new baseline, with remarkable results in Text-to-Image synthesis, among other fields. Aside its usefulness per se, it can also be particularly relevant as a tool for data augmentation to aid training models for other document image processing tasks. In this work, we present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles, by using class index styles and text content prompts without the need of adversarial training, writer recognition, or text recognition. We gauge system performance with the Fr\'echet Inception Distance, writer recognition accuracy, and writer retrieval. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data. Code is available at: https://github.com/koninik/WordStylist.

Related papers

• TypeScore: A Text Fidelity Metric for Text-to-Image Generative Models [39.06617653124486]
  We introduce a new evaluation framework called TypeScore to assess a model's ability to generate images with high-fidelity embedded text. Our proposed metric demonstrates greater resolution than CLIPScore to differentiate popular image generation models.
  arXiv  Detail & Related papers  (2024-11-02T07:56:54Z)
• Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation [67.89838237013078]
  Named entity recognition (NER) models often struggle with noisy inputs. We propose a more realistic setting in which only noisy text and its NER labels are available. We employ a multi-view training framework that improves robust NER without retrieving text during inference.
  arXiv  Detail & Related papers  (2024-07-26T07:30:41Z)
• Efficiently Leveraging Linguistic Priors for Scene Text Spotting [63.22351047545888]
  This paper proposes a method that leverages linguistic knowledge from a large text corpus to replace the traditional one-hot encoding used in auto-regressive scene text spotting and recognition models. We generate text distributions that align well with scene text datasets, removing the need for in-domain fine-tuning. Experimental results show that our method not only improves recognition accuracy but also enables more accurate localization of words.
  arXiv  Detail & Related papers  (2024-02-27T01:57:09Z)
• Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering [118.53208190209517]
  We propose a framework to learn the proper textual descriptions for diffusion models through prompt learning. Our method can effectively learn the prompts to improve the matches between the input text and the generated images.
  arXiv  Detail & Related papers  (2024-01-12T03:46:29Z)
• UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models [25.219960711604728]
  This paper proposes a novel approach for text image generation, utilizing a pre-trained diffusion model. Our approach involves the design and training of a light-weight character-level text encoder, which replaces the original CLIP encoder. By employing an inference stage refinement process, we achieve a notably high sequence accuracy when synthesizing text in arbitrarily given images.
  arXiv  Detail & Related papers  (2023-12-08T07:47:46Z)
• Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models [63.99110667987318]
  We present DiffText, a pipeline that seamlessly blends foreground text with the background's intrinsic features. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors.
  arXiv  Detail & Related papers  (2023-11-28T06:51:28Z)
• Style Generation: Image Synthesis based on Coarsely Matched Texts [10.939482612568433]
  We introduce a novel task called text-based style generation and propose a two-stage generative adversarial network. The first stage generates the overall image style with a sentence feature, and the second stage refines the generated style with a synthetic feature. The practical potential of our work is demonstrated by various applications such as text-image alignment and story visualization.
  arXiv  Detail & Related papers  (2023-09-08T21:51:11Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)
• Reading and Writing: Discriminative and Generative Modeling for Self-Supervised Text Recognition [101.60244147302197]
  We introduce contrastive learning and masked image modeling to learn discrimination and generation of text images. Our method outperforms previous self-supervised text recognition methods by 10.2%-20.2% on irregular scene text recognition datasets. Our proposed text recognizer exceeds previous state-of-the-art text recognition methods by averagely 5.3% on 11 benchmarks, with similar model size.
  arXiv  Detail & Related papers  (2022-07-01T03:50:26Z)
• Content and Style Aware Generation of Text-line Images for Handwriting Recognition [4.301658883577544]
  We propose a generative method for handwritten text-line images conditioned on both visual appearance and textual content. Our method is able to produce long text-line samples with diverse handwriting styles.
  arXiv  Detail & Related papers  (2022-04-12T05:52:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.