FonTS: Text Rendering with Typography and Style Controls

• URL: http://arxiv.org/abs/2412.00136v1
• Date: Thu, 28 Nov 2024 16:19:37 GMT
• Title: FonTS: Text Rendering with Typography and Style Controls
• Authors: Wenda Shi, Yiren Song, Dengming Zhang, Jiaming Liu, Xingxing Zou,
• Abstract summary: This paper proposes a two-stage DiT-based pipeline to enhance controllability over typography and style in text rendering. We introduce Typography Control (TC) finetuning, an efficient parameter fine-tuning method, and enclosing typography control tokens (ETC-tokens) To further enhance style control, we present a Style Control Adapter (SCA) that injects style information through image inputs independent of text prompts.
• Score: 12.717568891224074
• License:
• Abstract: Visual text images are prevalent in various applications, requiring careful font selection and typographic choices. Recent advances in Diffusion Transformer (DiT)-based text-to-image (T2I) models show promise in automating these processes. However, these methods still face challenges such as inconsistent fonts, style variation, and limited fine-grained control, particularly at the word level. This paper proposes a two-stage DiT-based pipeline to address these issues by enhancing controllability over typography and style in text rendering. We introduce Typography Control (TC) finetuning, an efficient parameter fine-tuning method, and enclosing typography control tokens (ETC-tokens), which enable precise word-level application of typographic features. To further enhance style control, we present a Style Control Adapter (SCA) that injects style information through image inputs independent of text prompts. Through comprehensive experiments, we demonstrate the effectiveness of our approach in achieving superior word-level typographic control, font consistency, and style consistency in Basic and Artistic Text Rendering (BTR and ATR) tasks. Our results mark a significant advancement in the precision and adaptability of T2I models, presenting new possibilities for creative applications and design-oriented tasks.

Related papers

• SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild [55.619708995575785]
  The text in natural scene images needs to meet the following four key criteria. The generated text can facilitate to the training of natural scene OCR (Optical Character Recognition) tasks. The generated images have superior utility in OCR tasks like text detection and text recognition.
  arXiv  Detail & Related papers  (2025-01-06T12:09:08Z)
• TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control [5.3798706094384725]
  We propose TextCtrl, a diffusion-based method that edits text with prior guidance control. Our method consists of two key components: (i) By constructing fine-grained text style disentanglement and robust text structure representation, TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. To fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons.
  arXiv  Detail & Related papers  (2024-10-14T03:50:39Z)
• FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation [19.65838242227773]
  This paper contributes a novel, concise, and efficient approach that adapts pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner. Our method allows flexible control over both guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band.
  arXiv  Detail & Related papers  (2024-08-02T04:13:38Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• Typographic Text Generation with Off-the-Shelf Diffusion Model [7.542892664684078]
  This paper proposes a typographic text generation system to add and modify text on typographic designs. The proposed system is a novel combination of two off-the-shelf methods for diffusion models, ControlNet and Blended Latent Diffusion.
  arXiv  Detail & Related papers  (2024-02-22T06:15:51Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• Improving Diffusion Models for Scene Text Editing with Dual Encoders [44.12999932588205]
  Scene text editing is a challenging task that involves modifying or inserting specified texts in an image. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. We propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design.
  arXiv  Detail & Related papers  (2023-04-12T02:08:34Z)
• Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors [58.71128866226768]
  Recent text-to-image generation methods have incrementally improved the generated image fidelity and text relevancy. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels.
  arXiv  Detail & Related papers  (2022-03-24T15:44:50Z)
• StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer [90.6768813620898]
  Style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. We introduce a large-scale benchmark, StylePTB, with paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text. We find that existing methods on StylePTB struggle to model fine-grained changes and have an even more difficult time composing multiple styles.
  arXiv  Detail & Related papers  (2021-04-12T04:25:09Z)
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery [71.1862388442953]
  We develop a text-based interface for StyleGAN image manipulation. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation.
  arXiv  Detail & Related papers  (2021-03-31T17:51:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.