FreeText: Training-Free Text Rendering in Diffusion Transformers via Attention Localization and Spectral Glyph Injection

• URL: http://arxiv.org/abs/2601.00535v1
• Date: Fri, 02 Jan 2026 02:36:48 GMT
• Title: FreeText: Training-Free Text Rendering in Diffusion Transformers via Attention Localization and Spectral Glyph Injection
• Abstract summary: Large-scale text-to-image (T2I) diffusion models excel at opendomain synthesis but still struggle with precise text rendering.<n>We propose textbfFreeText, a training-free, plug-and-play framework that improves text rendering by exploiting intrinsic mechanisms of emphDiffusion Transformer (DiT) models.
• Score: 28.229261085054745
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large-scale text-to-image (T2I) diffusion models excel at open-domain synthesis but still struggle with precise text rendering, especially for multi-line layouts, dense typography, and long-tailed scripts such as Chinese. Prior solutions typically require costly retraining or rigid external layout constraints, which can degrade aesthetics and limit flexibility. We propose \textbf{FreeText}, a training-free, plug-and-play framework that improves text rendering by exploiting intrinsic mechanisms of \emph{Diffusion Transformer (DiT)} models. \textbf{FreeText} decomposes the problem into \emph{where to write} and \emph{what to write}. For \emph{where to write}, we localize writing regions by reading token-wise spatial attribution from endogenous image-to-text attention, using sink-like tokens as stable spatial anchors and topology-aware refinement to produce high-confidence masks. For \emph{what to write}, we introduce Spectral-Modulated Glyph Injection (SGMI), which injects a noise-aligned glyph prior with frequency-domain band-pass modulation to strengthen glyph structure and suppress semantic leakage (rendering the concept instead of the word). Extensive experiments on Qwen-Image, FLUX.1-dev, and SD3 variants across longText-Benchmark, CVTG, and our CLT-Bench show consistent gains in text readability while largely preserving semantic alignment and aesthetic quality, with modest inference overhead.

Related papers

• GlyphAnchor: Enhancing Visual Text Rendering via Position-Anchored Glyph Priors [44.606371845449424]
  We introduce GlyphAnchor, a novel text-rendering enhancement method for both text-to-image and image-editing diffusion transformer models.<n> Experiments across multiple backbones and benchmarks, including long, complex, and densely arranged text and rare character scenarios, show that GlyphAnchor consistently improves text fidelity while preserving overall image quality.
  arXiv  Detail & Related papers  (2026-09-02T09:23:23Z)
• One Patch Is Enough: Reinforcement-Optimized Visual Token Grounding for MLLM-Based Scene Text Spotting [51.09550363815034]
  Scene text spotting requires high-precision alignment between textual recognition and spatial localization.<n>We propose Single-Patch Text Spotting (SPaTS), a vision-centric framework that routes each text instance through a single anchor visual token.<n>We show that SPaTS consistently and significantly outperforms both frontier closed-source MLLMs and OCR MLLMs.
  arXiv  Detail & Related papers  (2026-07-30T09:17:48Z)
• SteerVTE: Seamless Video Text Editing with Style and Glyph Control [37.44272702473038]
  Video text editing is a localized task demanding stroke-level precision within small text regions.<n>We introduce SteerVTE, a unified framework that underlinetextbfsteers a frozen video diffusion model.<n>We show that SteerVTE substantially outperforms existing video editing baselines across text accuracy, style consistency, and temporal coherence.
  arXiv  Detail & Related papers  (2026-06-22T12:37:15Z)
• Training-Free Occluded Text Rendering via Glyph Priors and Attention-Guided Semantic Blending [0.0]
  We present a training-free framework for occluded text rendering with a pretrained FLUX.1-dev backbone.<n>The task requires a model to render recognizable typography and place an occluding object over the intended text region.<n>We propose a restarted dual-stream inference framework that decouples text- preservation from occluder insertion.
  arXiv  Detail & Related papers  (2026-05-16T04:58:07Z)
• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• TextGround4M: A Prompt-Aligned Dataset for Layout-Aware Text Rendering [64.22226877213521]
  Despite advances in text-to-image generation, models still struggle to render prompt-specified text with correct spatial layout.<n>We introduce TextGround4M, a dataset of over 4 million prompt-image pairs annotated with span-level text grounded in the prompt and corresponding bounding boxes.<n>We propose a lightweight training strategy for autoregressive T2I models that appends layout-aware span tokens during training, without altering model architecture or inference behavior.
  arXiv  Detail & Related papers  (2026-04-27T13:28:57Z)
• TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment [68.91073792449201]
  We propose TextGuider, a training-free method that encourages accurate and complete text appearance.<n>Specifically, we analyze attention patterns in Multi-Modal Diffusion Transformer(MM-DiT) models, particularly for text-related tokens intended to be rendered in the image.<n>Our method achieves state-of-the-art performance in test-time text rendering, with significant gains in recall and strong results in OCR accuracy and CLIP score.
  arXiv  Detail & Related papers  (2025-12-10T06:18:30Z)
• UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis [38.658170067715965]
  We propose a segmentation-guided framework that uses pixel-level visual text masks as unified conditional inputs.<n>Our approach achieves state-of-the-art performance on the AnyText benchmark.<n>We also introduce two new benchmarks: GlyphMM-benchmark for testing layout and glyph consistency in complex, and MiniText-benchmark for assessing generation quality in small-scale text regions.
  arXiv  Detail & Related papers  (2025-07-01T17:42:19Z)
• GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing [23.64662356622401]
  We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision.<n>Our method achieves an 18.02% improvement in sentence accuracy over the state-of-the-art scene text editing baseline.
  arXiv  Detail & Related papers  (2025-05-08T03:11:58Z)
• FLUX-Text: A Simple and Advanced Diffusion Transformer Baseline for Scene Text Editing [15.626092015533072]
  Scene text editing aims to modify or add texts on images while ensuring text fidelity and overall visual quality consistent with the background.<n>Recent methods are primarily built on UNet-based diffusion models, which have improved scene text editing results, but still struggle with complex glyph structures.<n>We present textbfFLUX-Text, a simple and advanced multilingual scene text editing DiT method.
  arXiv  Detail & Related papers  (2025-05-06T08:56:28Z)
• RepText: Rendering Visual Text via Replicating [15.476598851383919]
  We present RepText, which aims to empower pre-trained monolingual text-to-image generation models with the ability to accurately render visual text in user-specified fonts.<n>Specifically, we adopt the setting from ControlNet and additionally integrate language agnostic glyph and position of rendered text to enable generating harmonized visual text.<n>Our approach outperforms existing open-source methods and achieves comparable results to native multi-language closed-source models.
  arXiv  Detail & Related papers  (2025-04-28T12:19:53Z)
• FTMoMamba: Motion Generation with Frequency and Text State Space Models [53.60865359814126]
  We propose a novel diffusion-based FTMoMamba framework equipped with a Frequency State Space Model and a Text State Space Model. To learn fine-grained representation, FreqSSM decomposes sequences into low-frequency and high-frequency components. To ensure the consistency between text and motion, TextSSM encodes text features at the sentence level.
  arXiv  Detail & Related papers  (2024-11-26T15:48:12Z)
• TextDiffuser: Diffusion Models as Text Painters [118.30923824681642]
  We introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. We contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs. We show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text.
  arXiv  Detail & Related papers  (2023-05-18T10:16:19Z)
• GlyphDiffusion: Text Generation as Image Generation [100.98428068214736]
  We propose GlyphDiffusion, a novel diffusion approach for text generation via text-guided image generation. Our key idea is to render the target text as a glyph image containing visual language content. Our model also makes significant improvements compared to the recent diffusion model.
  arXiv  Detail & Related papers  (2023-04-25T02:14:44Z)
• P+: Extended Textual Conditioning in Text-to-Image Generation [50.823884280133626]
  We introduce an Extended Textual Conditioning space in text-to-image models, referred to as $P+$. We show that the extended space provides greater disentangling and control over image synthesis. We further introduce Extended Textual Inversion (XTI), where the images are inverted into $P+$, and represented by per-layer tokens.
  arXiv  Detail & Related papers  (2023-03-16T17:38:15Z)
• All you need is a second look: Towards Tighter Arbitrary shape text detection [80.85188469964346]
  Long curve text instances tend to be fragmented because of the limited receptive field size of CNN. Simple representations using rectangle or quadrangle bounding boxes fall short when dealing with more challenging arbitrary-shaped texts. textitNASK reconstructs text instances with a more tighter representation using the predicted geometrical attributes.
  arXiv  Detail & Related papers  (2020-04-26T17:03:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.