TextPixs: Glyph-Conditioned Diffusion with Character-Aware Attention and OCR-Guided Supervision

• URL: http://arxiv.org/abs/2507.06033v1
• Date: Tue, 08 Jul 2025 14:35:02 GMT
• Title: TextPixs: Glyph-Conditioned Diffusion with Character-Aware Attention and OCR-Guided Supervision
• Abstract summary: This paper introduces a new framework, namely Glyph-Conditioned Diffusion with Character-Aware Attention.<n>It sets a new state-of-the-art on all metrics, with better character based metrics on text rendering.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The modern text-to-image diffusion models boom has opened a new era in digital content production as it has proven the previously unseen ability to produce photorealistic and stylistically diverse imagery based on the semantics of natural-language descriptions. However, the consistent disadvantage of these models is that they cannot generate readable, meaningful, and correctly spelled text in generated images, which significantly limits the use of practical purposes like advertising, learning, and creative design. This paper introduces a new framework, namely Glyph-Conditioned Diffusion with Character-Aware Attention (GCDA), using which a typical diffusion backbone is extended by three well-designed modules. To begin with, the model has a dual-stream text encoder that encodes both semantic contextual information and explicit glyph representations, resulting in a character-aware representation of the input text that is rich in nature. Second, an attention mechanism that is aware of the character is proposed with a new attention segregation loss that aims to limit the attention distribution of each character independently in order to avoid distortion artifacts. Lastly, GCDA has an OCR-in-the-loop fine-tuning phase, where a full text perceptual loss, directly optimises models to be legible and accurately spell. Large scale experiments to benchmark datasets, such as MARIO-10M and T2I-CompBench, reveal that GCDA sets a new state-of-the-art on all metrics, with better character based metrics on text rendering (Character Error Rate: 0.08 vs 0.21 for the previous best; Word Error Rate: 0.15 vs 0.25), human perception, and comparable image synthesis quality on high-fidelity (FID: 14.3).

Related papers

• STRICT: Stress Test of Rendering Images Containing Text [11.236527918747925]
  $textbfSTRICT$ is a benchmark designed to stress-test the ability of diffusion models to render coherent and instruction-aligned text in images.<n>We evaluate several state-of-the-art models, including proprietary and open-source variants, and reveal persistent limitations in long-range consistency and instruction-following capabilities.
  arXiv  Detail & Related papers  (2025-05-25T05:37:08Z)
• ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models [52.23899502520261]
  We introduce a novel framework named, ARTIST, which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically.<n>We finetune a visual diffusion model, enabling it to assimilate textual structure information from the pretrained textual model.<n>This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation.
  arXiv  Detail & Related papers  (2024-06-17T19:31:24Z)
• STAR: Scale-wise Text-conditioned AutoRegressive image generation [38.98271279816512]
  We introduce STAR, a text-to-image model that employs a scale-wise auto-regressive paradigm.<n> STAR enables text-driven image generation up to 1024$times$1024 through three key designs.
  arXiv  Detail & Related papers  (2024-06-16T03:45:45Z)
• Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation [5.55027585813848]
  The capability to generate visual text is crucial, offering both academic interest and a wide range of practical applications. We introduce a benchmark, LenCom-Eval, specifically designed for testing models' capability in generating images with Lengthy and Complex visual text. We demonstrate notable improvements across a range of evaluation metrics, including CLIPScore, OCR precision, recall, F1 score, accuracy, and edit distance scores.
  arXiv  Detail & Related papers  (2024-03-25T04:54:49Z)
• Contrastive Prompts Improve Disentanglement in Text-to-Image Diffusion Models [68.47333676663312]
  We show a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens. We illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.
  arXiv  Detail & Related papers  (2024-02-21T03:01:17Z)
• UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models [25.219960711604728]
  This paper proposes a novel approach for text image generation, utilizing a pre-trained diffusion model. Our approach involves the design and training of a light-weight character-level text encoder, which replaces the original CLIP encoder. By employing an inference stage refinement process, we achieve a notably high sequence accuracy when synthesizing text in arbitrarily given images.
  arXiv  Detail & Related papers  (2023-12-08T07:47:46Z)
• PV2TEA: Patching Visual Modality to Textual-Established Information Extraction [59.76117533540496]
  We patch the visual modality to the textual-established attribute information extractor. PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes. Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
  arXiv  Detail & Related papers  (2023-06-01T05:39:45Z)
• CoBIT: A Contrastive Bi-directional Image-Text Generation Model [72.1700346308106]
  CoBIT employs a novel unicoder-decoder structure, which attempts to unify three pre-training objectives in one framework. CoBIT achieves superior performance in image understanding, image-text understanding (Retrieval, Captioning, VQA, SNLI-VE) and text-based content creation, particularly in zero-shot scenarios.
  arXiv  Detail & Related papers  (2023-03-23T17:24:31Z)
• Self-supervised Character-to-Character Distillation for Text Recognition [54.12490492265583]
  We propose a novel self-supervised Character-to-Character Distillation method, CCD, which enables versatile augmentations to facilitate text representation learning. CCD achieves state-of-the-art results, with average performance gains of 1.38% in text recognition, 1.7% in text segmentation, 0.24 dB (PSNR) and 0.0321 (SSIM) in text super-resolution.
  arXiv  Detail & Related papers  (2022-11-01T05:48:18Z)
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation [95.02406834386814]
  Parti treats text-to-image generation as a sequence-to-sequence modeling problem. Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. PartiPrompts (P2) is a new holistic benchmark of over 1600 English prompts.
  arXiv  Detail & Related papers  (2022-06-22T01:11:29Z)
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding [53.170767750244366]
  Imagen is a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understanding. To assess text-to-image models in greater depth, we introduce DrawBench, a comprehensive and challenging benchmark for text-to-image models.
  arXiv  Detail & Related papers  (2022-05-23T17:42:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.