DreamText: High Fidelity Scene Text Synthesis

• URL: http://arxiv.org/abs/2405.14701v5
• Date: Mon, 24 Mar 2025 06:13:16 GMT
• Title: DreamText: High Fidelity Scene Text Synthesis
• Abstract summary: Scene text synthesis involves rendering specified texts onto arbitrary images.<n>Current methods typically formulate this task in an end-to-end manner but lack effective character-level guidance during training.<n>This paper proposes DreamText for high-fidelity scene text synthesis.
• Score: 14.257966899551212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scene text synthesis involves rendering specified texts onto arbitrary images. Current methods typically formulate this task in an end-to-end manner but lack effective character-level guidance during training. Besides, their text encoders, pre-trained on a single font type, struggle to adapt to the diverse font styles encountered in practical applications. Consequently, these methods suffer from character distortion, repetition, and absence, particularly in polystylistic scenarios. To this end, this paper proposes DreamText for high-fidelity scene text synthesis. Our key idea is to reconstruct the diffusion training process, introducing more refined guidance tailored to this task, to expose and rectify the model's attention at the character level and strengthen its learning of text regions. This transformation poses a hybrid optimization challenge, involving both discrete and continuous variables. To effectively tackle this challenge, we employ a heuristic alternate optimization strategy. Meanwhile, we jointly train the text encoder and generator to comprehensively learn and utilize the diverse font present in the training dataset. This joint training is seamlessly integrated into the alternate optimization process, fostering a synergistic relationship between learning character embedding and re-estimating character attention. Specifically, in each step, we first encode potential character-generated position information from cross-attention maps into latent character masks. These masks are then utilized to update the representation of specific characters in the current step, which, in turn, enables the generator to correct the character's attention in the subsequent steps. Both qualitative and quantitative results demonstrate the superiority of our method to the state of the art.

Related papers

• Towards Training-Free Scene Text Editing [14.724015832146463]
  Scene text editing seeks to modify textual content in natural images.<n>Existing methods often require task-specific training or paired data.<n>We propose TextFlow, a training-free scene text editing framework.
  arXiv  Detail & Related papers  (2026-03-25T17:50:31Z)
• LogoDiffuser: Training-Free Multilingual Logo Generation and Stylization via Letter-Aware Attention Control [6.75235033060142]
  We propose a training-free method that synthesizes multilingual logo designs using the multimodal diffusion transformer.<n>Instead of using textual prompts, we input the target characters as images, enabling robust character structure control regardless of language.<n>Our method achieves state-of-the-art performance in multilingual logo generation.
  arXiv  Detail & Related papers  (2026-03-10T14:57:46Z)
• Unveiling Text in Challenging Stone Inscriptions: A Character-Context-Aware Patching Strategy for Binarization [7.572552907946751]
  Binarization is a popular first step towards text extraction in historical artifacts.<n>We present a robust and adaptive patching strategy to binarize challenging Indic inscriptions.<n>We demonstrate that our novel patching mechanism significantly boosts binarization performance across classical and deep learning baselines.
  arXiv  Detail & Related papers  (2026-01-07T05:37:29Z)
• TextGuider: Training-Free Guidance for Text Rendering via Attention Alignment [68.91073792449201]
  We propose TextGuider, a training-free method that encourages accurate and complete text appearance.<n>Specifically, we analyze attention patterns in Multi-Modal Diffusion Transformer(MM-DiT) models, particularly for text-related tokens intended to be rendered in the image.<n>Our method achieves state-of-the-art performance in test-time text rendering, with significant gains in recall and strong results in OCR accuracy and CLIP score.
  arXiv  Detail & Related papers  (2025-12-10T06:18:30Z)
• GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing [23.64662356622401]
  We present GlyphMastero, a specialized glyph encoder designed to guide the latent diffusion model for generating texts with stroke-level precision.<n>Our method achieves an 18.02% improvement in sentence accuracy over the state-of-the-art scene text editing baseline.
  arXiv  Detail & Related papers  (2025-05-08T03:11:58Z)
• Efficient Personalized Text-to-image Generation by Leveraging Textual Subspace [52.24866347353916]
  We propose an efficient method to explore the target embedding in a textual subspace. We also propose an efficient selection strategy for determining the basis of the textual subspace. Our method opens the door to more efficient representation learning for personalized text-to-image generation.
  arXiv  Detail & Related papers  (2024-06-30T06:41:21Z)
• UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models [25.219960711604728]
  This paper proposes a novel approach for text image generation, utilizing a pre-trained diffusion model. Our approach involves the design and training of a light-weight character-level text encoder, which replaces the original CLIP encoder. By employing an inference stage refinement process, we achieve a notably high sequence accuracy when synthesizing text in arbitrarily given images.
  arXiv  Detail & Related papers  (2023-12-08T07:47:46Z)
• Style Generation: Image Synthesis based on Coarsely Matched Texts [10.939482612568433]
  We introduce a novel task called text-based style generation and propose a two-stage generative adversarial network. The first stage generates the overall image style with a sentence feature, and the second stage refines the generated style with a synthetic feature. The practical potential of our work is demonstrated by various applications such as text-image alignment and story visualization.
  arXiv  Detail & Related papers  (2023-09-08T21:51:11Z)
• Text-Only Training for Visual Storytelling [107.19873669536523]
  We formulate visual storytelling as a visual-conditioned story generation problem. We propose a text-only training method that separates the learning of cross-modality alignment and story generation.
  arXiv  Detail & Related papers  (2023-08-17T09:32:17Z)
• MSSRNet: Manipulating Sequential Style Representation for Unsupervised Text Style Transfer [82.37710853235535]
  Unsupervised text style transfer task aims to rewrite a text into target style while preserving its main content. Traditional methods rely on the use of a fixed-sized vector to regulate text style, which is difficult to accurately convey the style strength for each individual token. Our proposed method addresses this issue by assigning individual style vector to each token in a text, allowing for fine-grained control and manipulation of the style strength.
  arXiv  Detail & Related papers  (2023-06-12T13:12:29Z)
• Improving Diffusion Models for Scene Text Editing with Dual Encoders [44.12999932588205]
  Scene text editing is a challenging task that involves modifying or inserting specified texts in an image. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. We propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design.
  arXiv  Detail & Related papers  (2023-04-12T02:08:34Z)
• Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Training [178.09150600453205]
  In this paper, we investigate offline open-vocabulary text-to-motion generation in a zero-shot learning manner. Inspired by the prompt learning in NLP, we pretrain a motion generator that learns to reconstruct the full motion from the masked motion. Our method reformulates the input text into a masked motion as the prompt for the motion generator to reconstruct'' the motion.
  arXiv  Detail & Related papers  (2022-10-28T06:20:55Z)
• Text Revision by On-the-Fly Representation Optimization [76.11035270753757]
  Current state-of-the-art methods formulate these tasks as sequence-to-sequence learning problems. We present an iterative in-place editing approach for text revision, which requires no parallel data. It achieves competitive and even better performance than state-of-the-art supervised methods on text simplification.
  arXiv  Detail & Related papers  (2022-04-15T07:38:08Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)
• RewriteNet: Realistic Scene Text Image Generation via Editing Text in Real-world Image [17.715320405808935]
  Scene text editing (STE) is a challenging task due to a complex intervention between text and style. We propose a novel representational learning-based STE model, referred to as RewriteNet. Our experiments demonstrate that RewriteNet achieves better quantitative and qualitative performance than other comparisons.
  arXiv  Detail & Related papers  (2021-07-23T06:32:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.