Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech

• URL: http://arxiv.org/abs/2606.14750v1
• Date: Fri, 05 Jun 2026 11:46:30 GMT
• Title: Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech
• Abstract summary: Grounding text in its visual form allows structurally similar characters with different Unicode encodings to produce similar embeddings.<n>We propose Pixel-TTS, the first framework for visually grounded speech synthesis.<n>It renders text as images and projects them through a 2D convolutional layer to generate embeddings.
• Score: 3.62994537177299
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding. Grounding text in its visual form allows structurally similar characters with different Unicode encodings to produce similar embeddings, benefiting cross-lingual and zero-shot scenarios. Conventional text-based approaches treat each character independently, limiting generalization to unseen characters and requiring embedding expansion during cross-lingual adaptation. We propose Pixel-TTS, the first framework for visually grounded speech synthesis. It renders text as images and projects them through a 2D convolutional layer to generate embeddings. This design eliminates embedding matrix expansion during fine-tuning while improving robustness to unseen characters and orthographic variations. Extensive experiments show Pixel-TTS achieves competitive performance with strong baselines, faster convergence and robust zero-shot generalization.

Related papers

• GlyphAnchor: Enhancing Visual Text Rendering via Position-Anchored Glyph Priors [44.606371845449424]
  We introduce GlyphAnchor, a novel text-rendering enhancement method for both text-to-image and image-editing diffusion transformer models.<n> Experiments across multiple backbones and benchmarks, including long, complex, and densely arranged text and rare character scenarios, show that GlyphAnchor consistently improves text fidelity while preserving overall image quality.
  arXiv  Detail & Related papers  (2026-09-02T09:23:23Z)
• TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment [68.91073792449201]
  We propose TextGuider, a training-free method that encourages accurate and complete text appearance.<n>Specifically, we analyze attention patterns in Multi-Modal Diffusion Transformer(MM-DiT) models, particularly for text-related tokens intended to be rendered in the image.<n>Our method achieves state-of-the-art performance in test-time text rendering, with significant gains in recall and strong results in OCR accuracy and CLIP score.
  arXiv  Detail & Related papers  (2025-12-10T06:18:30Z)
• UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation [51.31795451147935]
  We present a unified generative model that supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework.<n>Our goal is to achieve unification along three axes: the model, the tasks, and the representations.<n> Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment.
  arXiv  Detail & Related papers  (2025-11-21T03:02:10Z)
• Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach [51.95266411355865]
  Autoregressive language models are vulnerable to orthographic attacks.<n>This vulnerability stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings.<n>We propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images.
  arXiv  Detail & Related papers  (2025-08-28T20:48:38Z)
• UniGlyph: Unified Segmentation-Conditioned Diffusion for Precise Visual Text Synthesis [38.658170067715965]
  We propose a segmentation-guided framework that uses pixel-level visual text masks as unified conditional inputs.<n>Our approach achieves state-of-the-art performance on the AnyText benchmark.<n>We also introduce two new benchmarks: GlyphMM-benchmark for testing layout and glyph consistency in complex, and MiniText-benchmark for assessing generation quality in small-scale text regions.
  arXiv  Detail & Related papers  (2025-07-01T17:42:19Z)
• EasyText: Controllable Diffusion Transformer for Multilingual Text Rendering [9.087419148444225]
  This paper introduces EasyText, a text rendering framework based on DiT (Diffusion Transformer)<n>We propose character positioning encoding and position encoding techniques to achieve controllable and precise text rendering.<n>We construct a large-scale synthetic text image dataset with 1 million multilingual image-text annotations as well as a high-quality dataset of 20K annotated images.
  arXiv  Detail & Related papers  (2025-05-30T09:55:39Z)
• TextSR: Diffusion Super-Resolution with Multilingual OCR Guidance [24.242452422416438]
  We introduce TextSR, a multimodal diffusion model specifically designed for Multilingual Text Image Super-Resolution.<n>By integrating text character priors with the low-resolution text images, our model effectively guides the super-resolution process.<n>The superior performance of our model on both the TextZoom and TextVQA datasets sets a new benchmark for STISR.
  arXiv  Detail & Related papers  (2025-05-29T05:40:35Z)
• Story Visualization by Online Text Augmentation with Context Memory [64.86944645907771]
  We propose a novel memory architecture for the Bi-directional Transformer framework with an online text augmentation. The proposed method significantly outperforms the state of the arts in various metrics including FID, character F1, frame accuracy, BLEU-2/3, and R-precision.
  arXiv  Detail & Related papers  (2023-08-15T05:08:12Z)
• GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generation [18.396131717250793]
  We introduce GlyphDraw, a general learning framework aiming to endow image generation models with the capacity to generate images coherently embedded with text for any specific language. Our method not only produces accurate language characters as in prompts, but also seamlessly blends the generated text into the background.
  arXiv  Detail & Related papers  (2023-03-31T08:06:33Z)
• CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
  We propose an end-to-end CLIP-Driven Referring Image framework (CRIS) CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment. Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
  arXiv  Detail & Related papers  (2021-11-30T07:29:08Z)
• TediGAN: Text-Guided Diverse Face Image Generation and Manipulation [52.83401421019309]
  TediGAN is a framework for multi-modal image generation and manipulation with textual descriptions. StyleGAN inversion module maps real images to the latent space of a well-trained StyleGAN. visual-linguistic similarity learns the text-image matching by mapping the image and text into a common embedding space. instance-level optimization is for identity preservation in manipulation.
  arXiv  Detail & Related papers  (2020-12-06T16:20:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.