Diffusion models for Handwriting Generation

• URL: http://arxiv.org/abs/2011.06704v1
• Date: Fri, 13 Nov 2020 00:31:22 GMT
• Title: Diffusion models for Handwriting Generation
• Abstract summary: We propose a diffusion probabilistic model for handwriting generation. Our method does not require using any text-recognition based, writer-style based, or adversarial loss functions. Our model is able to incorporate writer stylistic features directly from image data, eliminating the need for user interaction during sampling.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In this paper, we propose a diffusion probabilistic model for handwriting generation. Diffusion models are a class of generative models where samples start from Gaussian noise and are gradually denoised to produce output. Our method of handwriting generation does not require using any text-recognition based, writer-style based, or adversarial loss functions, nor does it require training of auxiliary networks. Our model is able to incorporate writer stylistic features directly from image data, eliminating the need for user interaction during sampling. Experiments reveal that our model is able to generate realistic , high quality images of handwritten text in a similar style to a given writer. Our implementation can be found at https://github.com/tcl9876/Diffusion-Handwriting-Generation

Related papers

• Diffusion Self-Distillation for Zero-Shot Customized Image Generation [40.11194010431839]
  Diffusion Self-Distillation is a method for generating its own dataset for text-conditioned image-to-image tasks. We first leverage a text-to-image diffusion model's in-context generation ability to create grids of images. We then fine-tune the text-to-image model into a text+image-to-image model using the curated paired dataset.
  arXiv  Detail & Related papers  (2024-11-27T18:58:52Z)
• Prompt-Propose-Verify: A Reliable Hand-Object-Interaction Data Generation Framework using Foundational Models [0.0]
  Diffusion models when conditioned on text prompts, generate realistic-looking images with intricate details. But most of these pre-trained models fail to generate accurate images when it comes to human features like hands, teeth, etc.
  arXiv  Detail & Related papers  (2023-12-23T12:59:22Z)
• Generative Pre-training for Speech with Flow Matching [81.59952572752248]
  We pre-trained a generative model, named SpeechFlow, on 60k hours of untranscribed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separation, and synthesis.
  arXiv  Detail & Related papers  (2023-10-25T03:40:50Z)
• Reverse Stable Diffusion: What prompt was used to generate this image? [73.10116197883303]
  We study the task of predicting the prompt embedding given an image generated by a generative diffusion model. We propose a novel learning framework comprising a joint prompt regression and multi-label vocabulary classification objective. We conduct experiments on the DiffusionDB data set, predicting text prompts from images generated by Stable Diffusion.
  arXiv  Detail & Related papers  (2023-08-02T23:39:29Z)
• Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners [88.07317175639226]
  We propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach mainly uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information.
  arXiv  Detail & Related papers  (2023-05-18T05:41:36Z)
• GlyphDiffusion: Text Generation as Image Generation [100.98428068214736]
  We propose GlyphDiffusion, a novel diffusion approach for text generation via text-guided image generation. Our key idea is to render the target text as a glyph image containing visual language content. Our model also makes significant improvements compared to the recent diffusion model.
  arXiv  Detail & Related papers  (2023-04-25T02:14:44Z)
• WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models [8.334487584550185]
  We present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data.
  arXiv  Detail & Related papers  (2023-03-29T10:19:26Z)
• Text-to-Image Diffusion Models are Zero-Shot Classifiers [8.26990105697146]
  We investigate text-to-image diffusion models by proposing a method for evaluating them as zero-shot classifiers. We apply our method to Stable Diffusion and Imagen, using it to probe fine-grained aspects of the models' knowledge. They perform competitively with CLIP on a wide range of zero-shot image classification datasets.
  arXiv  Detail & Related papers  (2023-03-27T14:15:17Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)
• DiffusER: Discrete Diffusion via Edit-based Reconstruction [88.62707047517914]
  DiffusER is an edit-based generative model for text based on denoising diffusion models. It can rival autoregressive models on several tasks spanning machine translation, summarization, and style transfer. It can also perform other varieties of generation that standard autoregressive models are not well-suited for.
  arXiv  Detail & Related papers  (2022-10-30T16:55:23Z)
• What is the Reward for Handwriting? -- Handwriting Generation by Imitation Learning [12.9611801481722]
  We use a reinforcement learning framework to realize handwriting generation with the careful future planning ability. For the algorithm, we employ generative adversarial imitation learning (GAIL) In other words, through GAIL, we can understand the reward of the handwriting generation process from handwriting examples.
  arXiv  Detail & Related papers  (2020-09-23T07:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.