EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering

• URL: http://arxiv.org/abs/2505.24417v1
• Date: Fri, 30 May 2025 09:55:39 GMT
• Title: EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering
• Authors: Runnan Lu, Yuxuan Zhang, Jailing Liu, Haifa Wang, Yiren Song,
• Abstract summary: This paper introduces EasyText, a text rendering framework based on DiT (Diffusion Transformer)<n>We propose character positioning encoding and position encoding techniques to achieve controllable and precise text rendering.<n>We construct a large-scale synthetic text image dataset with 1 million multilingual image-text annotations as well as a high-quality dataset of 20K annotated images.
• Score: 9.087419148444225
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating accurate multilingual text with diffusion models has long been desired but remains challenging. Recent methods have made progress in rendering text in a single language, but rendering arbitrary languages is still an unexplored area. This paper introduces EasyText, a text rendering framework based on DiT (Diffusion Transformer), which connects denoising latents with multilingual character tokens encoded as character tokens. We propose character positioning encoding and position encoding interpolation techniques to achieve controllable and precise text rendering. Additionally, we construct a large-scale synthetic text image dataset with 1 million multilingual image-text annotations as well as a high-quality dataset of 20K annotated images, which are used for pretraining and fine-tuning respectively. Extensive experiments and evaluations demonstrate the effectiveness and advancement of our approach in multilingual text rendering, visual quality, and layout-aware text integration.

Related papers

• UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis [38.658170067715965]
  We propose a segmentation-guided framework that uses pixel-level visual text masks as unified conditional inputs.<n>Our approach achieves state-of-the-art performance on the AnyText benchmark.<n>We also introduce two new benchmarks: GlyphMM-benchmark for testing layout and glyph consistency in complex, and MiniText-benchmark for assessing generation quality in small-scale text regions.
  arXiv  Detail & Related papers  (2025-07-01T17:42:19Z)
• RepText: Rendering Visual Text via Replicating [15.476598851383919]
  We present RepText, which aims to empower pre-trained monolingual text-to-image generation models with the ability to accurately render visual text in user-specified fonts.<n>Specifically, we adopt the setting from ControlNet and additionally integrate language agnostic glyph and position of rendered text to enable generating harmonized visual text.<n>Our approach outperforms existing open-source methods and achieves comparable results to native multi-language closed-source models.
  arXiv  Detail & Related papers  (2025-04-28T12:19:53Z)
• Multi-language Video Subtitle Dataset for Image-based Text Recognition [0.0]
  This dataset includes 4,224 subtitle images extracted from 24 videos sourced from online platforms. It features a wide variety of characters, including Thai consonants, vowels, tone marks, punctuation marks, numerals, Roman characters, and Arabic numerals.
  arXiv  Detail & Related papers  (2024-11-07T00:06:53Z)
• AnyTrans: Translate AnyText in the Image with Large Scale Models [88.5887934499388]
  This paper introduces AnyTrans, an all-encompassing framework for the task-Translate AnyText in the Image (TATI) Our framework incorporates contextual cues from both textual and visual elements during translation. We have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.
  arXiv  Detail & Related papers  (2024-06-17T11:37:48Z)
• Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models [63.99110667987318]
  We present DiffText, a pipeline that seamlessly blends foreground text with the background's intrinsic features. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors.
  arXiv  Detail & Related papers  (2023-11-28T06:51:28Z)
• TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering [118.30923824681642]
  TextDiffuser-2 aims to unleash the power of language models for text rendering. We utilize the language model within the diffusion model to encode the position and texts at the line level. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V.
  arXiv  Detail & Related papers  (2023-11-28T04:02:40Z)
• AnyText: Multilingual Visual Text Generation And Editing [18.811943975513483]
  We introduce AnyText, a diffusion-based multilingual visual text generation and editing model. AnyText can write characters in multiple languages, to the best of our knowledge, this is the first work to address multilingual visual text generation. We contribute the first large-scale multilingual text images dataset, AnyWord-3M, containing 3 million image-text pairs with OCR annotations in multiple languages.
  arXiv  Detail & Related papers  (2023-11-06T12:10:43Z)
• TextDiffuser: Diffusion Models as Text Painters [118.30923824681642]
  We introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. We contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs. We show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text.
  arXiv  Detail & Related papers  (2023-05-18T10:16:19Z)
• Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation [63.061871048769596]
  We present a novel Unified Multi-Modal Latent Diffusion (UMM-Diffusion) which takes joint texts and images containing specified subjects as input sequences. To be more specific, both input texts and images are encoded into one unified multi-modal latent space. Our method is able to generate high-quality images with complex semantics from both aspects of input texts and images.
  arXiv  Detail & Related papers  (2023-03-16T13:50:20Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.