CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control

• URL: http://arxiv.org/abs/2410.03787v1
• Date: Thu, 3 Oct 2024 20:26:54 GMT
• Title: CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control
• Authors: Qisheng Liao, Liang Li, Yulang Fei, Gus Xia,
• Abstract summary: CalliffusionV2 is a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. It excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach. It is also capable of generating non-Chinese characters without prior training.
• Score: 8.2481475383203
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce CalliffusionV2, a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. Unlike previous approaches that rely solely on image or text inputs and lack fine-grained control, our system leverages both images to guide generations at fine-grained levels and natural language texts to describe the features of generations. CalliffusionV2 excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach. It is also capable of generating non-Chinese characters without prior training. Comprehensive tests confirm that our system produces calligraphy that is both stylistically accurate and recognizable by neural network classifiers and human evaluators.

Related papers

• Decoupling Layout from Glyph in Online Chinese Handwriting Generation [6.566541829858544]
  We develop a text line layout generator and stylized font synthesizer. The layout generator performs in-context-like learning based on the text content and the provided style references to generate positions for each glyph autoregressively. The font synthesizer which consists of a character embedding dictionary, a multi-scale calligraphy style encoder, and a 1D U-Net based diffusion denoiser will generate each font on its position while imitating the calligraphy style extracted from the given style references.
  arXiv  Detail & Related papers  (2024-10-03T08:46:17Z)
• DeepCalliFont: Few-shot Chinese Calligraphy Font Synthesis by Integrating Dual-modality Generative Models [20.76773399161289]
  Few-shot font generation, especially for Chinese calligraphy fonts, is a challenging and ongoing problem. We propose a novel model, DeepCalliFont, for few-shot Chinese calligraphy font synthesis by integrating dual-modality generative models.
  arXiv  Detail & Related papers  (2023-12-16T04:23:12Z)
• Calliffusion: Chinese Calligraphy Generation and Style Transfer with Diffusion Modeling [1.856334276134661]
  We propose Calliffusion, a system for generating high-quality Chinese calligraphy using diffusion models. Our model architecture is based on DDPM (Denoising Diffusion Probabilistic Models)
  arXiv  Detail & Related papers  (2023-05-30T15:34:45Z)
• GlyphDiffusion: Text Generation as Image Generation [100.98428068214736]
  We propose GlyphDiffusion, a novel diffusion approach for text generation via text-guided image generation. Our key idea is to render the target text as a glyph image containing visual language content. Our model also makes significant improvements compared to the recent diffusion model.
  arXiv  Detail & Related papers  (2023-04-25T02:14:44Z)
• Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation [63.061871048769596]
  We present a novel Unified Multi-Modal Latent Diffusion (UMM-Diffusion) which takes joint texts and images containing specified subjects as input sequences. To be more specific, both input texts and images are encoded into one unified multi-modal latent space. Our method is able to generate high-quality images with complex semantics from both aspects of input texts and images.
  arXiv  Detail & Related papers  (2023-03-16T13:50:20Z)
• Universal Multimodal Representation for Language Understanding [110.98786673598015]
  This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively.
  arXiv  Detail & Related papers  (2023-01-09T13:54:11Z)
• Controllable Natural Language Generation with Contrastive Prefixes [120.12778570283956]
  GPT2 generation utilizes a set of small attribute-specific vectors, called prefixes, to steer natural language generation. We propose a novel supervised method and also an unsupervised method to train the prefixes for single-aspect control. Experimental results on both single-aspect and multi-aspect control show that our methods can guide generation towards the desired attributes while keeping high linguistic quality.
  arXiv  Detail & Related papers  (2022-02-27T00:31:03Z)
• Scalable Font Reconstruction with Dual Latent Manifolds [55.29525824849242]
  We propose a deep generative model that performs typography analysis and font reconstruction. Our approach enables us to massively scale up the number of character types we can effectively model. We evaluate on the task of font reconstruction over various datasets representing character types of many languages.
  arXiv  Detail & Related papers  (2021-09-10T20:37:43Z)
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance [99.68851329919858]
  discrete nature of natural language makes disentangling of textual representations more challenging. Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation.
  arXiv  Detail & Related papers  (2020-06-01T03:36:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.