FonTS: Text Rendering with Typography and Style Controls

• URL: http://arxiv.org/abs/2412.00136v2
• Date: Mon, 10 Mar 2025 08:43:03 GMT
• Title: FonTS: Text Rendering with Typography and Style Controls
• Authors: Wenda Shi, Yiren Song, Dengming Zhang, Jiaming Liu, Xingxing Zou,
• Abstract summary: This paper proposes a two-stage DiT-based pipeline to address problems by enhancing controllability over typography and style in text rendering.<n>We introduce typography control fine-tuning (TC-FT), an parameter-efficient fine-tuning method with enclosing typography control tokens (ETC-tokens)<n>To further address style inconsistency in text rendering, we propose a text-agnostic style control adapter (SCA) that prevents content leakage while enhancing style consistency.
• Score: 12.717568891224074
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual text rendering are widespread in various real-world applications, requiring careful font selection and typographic choices. Recent progress in diffusion transformer (DiT)-based text-to-image (T2I) models show promise in automating these processes. However, these methods still encounter challenges like inconsistent fonts, style variation, and limited fine-grained control, particularly at the word-level. This paper proposes a two-stage DiT-based pipeline to address these problems by enhancing controllability over typography and style in text rendering. We introduce typography control fine-tuning (TC-FT), an parameter-efficient fine-tuning method (on $5\%$ key parameters) with enclosing typography control tokens (ETC-tokens), which enables precise word-level application of typographic features. To further address style inconsistency in text rendering, we propose a text-agnostic style control adapter (SCA) that prevents content leakage while enhancing style consistency. To implement TC-FT and SCA effectively, we incorporated HTML-render into the data synthesis pipeline and proposed the first word-level controllable dataset. Through comprehensive experiments, we demonstrate the effectiveness of our approach in achieving superior word-level typographic control, font consistency, and style consistency in text rendering tasks. The datasets and models will be available for academic use.

Related papers

• WordCon: Word-level Typography Control in Scene Text Rendering [12.00205643907721]
  We construct a word-level controlled scene text dataset and introduce the Text-Image Alignment framework.<n>We also propose WordCon, a hybrid parameter-efficient fine-tuning (PEFT) method.<n>The datasets and source code will be available for academic use.
  arXiv  Detail & Related papers  (2025-06-26T14:00:38Z)
• SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild [55.619708995575785]
  The text in natural scene images needs to meet the following four key criteria. The generated text can facilitate to the training of natural scene OCR (Optical Character Recognition) tasks. The generated images have superior utility in OCR tasks like text detection and text recognition.
  arXiv  Detail & Related papers  (2025-01-06T12:09:08Z)
• TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control [5.3798706094384725]
  We propose TextCtrl, a diffusion-based method that edits text with prior guidance control. Our method consists of two key components: (i) By constructing fine-grained text style disentanglement and robust text structure representation, TextCtrl explicitly incorporates Style-Structure guidance into model design and network training, significantly improving text style consistency and rendering accuracy. To fill the vacancy of the real-world STE evaluation benchmark, we create the first real-world image-pair dataset termed ScenePair for fair comparisons.
  arXiv  Detail & Related papers  (2024-10-14T03:50:39Z)
• FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation [19.65838242227773]
  This paper contributes a novel, concise, and efficient approach that adapts pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner. Our method allows flexible control over both guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band.
  arXiv  Detail & Related papers  (2024-08-02T04:13:38Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• Typographic Text Generation with Off-the-Shelf Diffusion Model [7.542892664684078]
  This paper proposes a typographic text generation system to add and modify text on typographic designs. The proposed system is a novel combination of two off-the-shelf methods for diffusion models, ControlNet and Blended Latent Diffusion.
  arXiv  Detail & Related papers  (2024-02-22T06:15:51Z)
• UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models [25.219960711604728]
  This paper proposes a novel approach for text image generation, utilizing a pre-trained diffusion model. Our approach involves the design and training of a light-weight character-level text encoder, which replaces the original CLIP encoder. By employing an inference stage refinement process, we achieve a notably high sequence accuracy when synthesizing text in arbitrarily given images.
  arXiv  Detail & Related papers  (2023-12-08T07:47:46Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering [118.30923824681642]
  TextDiffuser-2 aims to unleash the power of language models for text rendering. We utilize the language model within the diffusion model to encode the position and texts at the line level. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V.
  arXiv  Detail & Related papers  (2023-11-28T04:02:40Z)
• Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain [53.22419717434372]
  We propose a new task, namely stylized data-to-text generation, whose aim is to generate coherent text according to a specific style. This task is non-trivial, due to three challenges: the logic of the generated text, unstructured style reference, and biased training samples. We propose a novel stylized data-to-text generation model, named StyleD2T, comprising three components: logic planning-enhanced data embedding, mask-based style embedding, and unbiased stylized text generation.
  arXiv  Detail & Related papers  (2023-05-05T03:02:41Z)
• Improving Diffusion Models for Scene Text Editing with Dual Encoders [44.12999932588205]
  Scene text editing is a challenging task that involves modifying or inserting specified texts in an image. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. We propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design.
  arXiv  Detail & Related papers  (2023-04-12T02:08:34Z)
• XMP-Font: Self-Supervised Cross-Modality Pre-training for Few-Shot Font Generation [13.569449355929574]
  We propose a self-supervised cross-modality pre-training strategy and a cross-modality transformer-based encoder. The encoder is conditioned jointly on the glyph image and the corresponding stroke labels. It only requires one reference glyph and achieves the lowest rate of bad cases in the few-shot font generation task 28% lower than the second best.
  arXiv  Detail & Related papers  (2022-04-11T13:34:40Z)
• Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors [58.71128866226768]
  Recent text-to-image generation methods have incrementally improved the generated image fidelity and text relevancy. We propose a novel text-to-image method that addresses these gaps by (i) enabling a simple control mechanism complementary to text in the form of a scene. Our model achieves state-of-the-art FID and human evaluation results, unlocking the ability to generate high fidelity images in a resolution of 512x512 pixels.
  arXiv  Detail & Related papers  (2022-03-24T15:44:50Z)
• Fine-grained style control in Transformer-based Text-to-speech Synthesis [78.92428622630861]
  We present a novel architecture to realize fine-grained style control on the Transformer-based text-to-speech synthesis (TransformerTTS) We model the speaking style by extracting a time sequence of local style tokens (LST) from the reference speech. Experiments show that with fine-grained style control, our system performs better in terms of naturalness, intelligibility, and style transferability.
  arXiv  Detail & Related papers  (2021-10-12T19:50:02Z)
• StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfer [90.6768813620898]
  Style transfer aims to controllably generate text with targeted stylistic changes while maintaining core meaning from the source sentence constant. We introduce a large-scale benchmark, StylePTB, with paired sentences undergoing 21 fine-grained stylistic changes spanning atomic lexical, syntactic, semantic, and thematic transfers of text. We find that existing methods on StylePTB struggle to model fine-grained changes and have an even more difficult time composing multiple styles.
  arXiv  Detail & Related papers  (2021-04-12T04:25:09Z)
• StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery [71.1862388442953]
  We develop a text-based interface for StyleGAN image manipulation. We first introduce an optimization scheme that utilizes a CLIP-based loss to modify an input latent vector in response to a user-provided text prompt. Next, we describe a latent mapper that infers a text-guided latent manipulation step for a given input image, allowing faster and more stable text-based manipulation.
  arXiv  Detail & Related papers  (2021-03-31T17:51:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.