FontUse: A Data-Centric Approach to Style- and Use-Case-Conditioned In-Image Typography

• URL: http://arxiv.org/abs/2603.06038v1
• Date: Fri, 06 Mar 2026 08:47:50 GMT
• Title: FontUse: A Data-Centric Approach to Style- and Use-Case-Conditioned In-Image Typography
• Abstract summary: We train image generation models using targeted supervision derived from a structured annotation pipeline specialized for typography.<n>Our pipeline constructs a large-scale typography-focused dataset, FontUse, consisting of about 70K images annotated with user-friendly prompts.<n>For evaluation, we introduce a Long-CLIP-based metric that measures alignment between generated typography and requested attributes.
• Score: 5.862480696321742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent text-to-image models can generate high-quality images from natural-language prompts, yet controlling typography remains challenging: requested typographic appearance is often ignored or only weakly followed. We address this limitation with a data-centric approach that trains image generation models using targeted supervision derived from a structured annotation pipeline specialized for typography. Our pipeline constructs a large-scale typography-focused dataset, FontUse, consisting of about 70K images annotated with user-friendly prompts, text-region locations, and OCR-recognized strings. The annotations are automatically produced using segmentation models and multimodal large language models (MLLMs). The prompts explicitly combine font styles (e.g., serif, script, elegant) and use cases (e.g., wedding invitations, coffee-shop menus), enabling intuitive specification even for novice users. Fine-tuning existing generators with these annotations allows them to consistently interpret style and use-case conditions as textual prompts without architectural modification. For evaluation, we introduce a Long-CLIP-based metric that measures alignment between generated typography and requested attributes. Experiments across diverse prompts and layouts show that models trained with our pipeline produce text renderings more consistent with prompts than competitive baselines. The source code for our annotation pipeline is available at https://github.com/xiaxinz/FontUSE.

Related papers

• Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions [33.440447854396446]
  We train the first open-source text-to-image model on long structured captions.<n>To process long captions efficiently, we propose DimFusion.<n>We also introduce the Text-as-a-Bottleneck Reconstruction (TaBR) evaluation protocol.
  arXiv  Detail & Related papers  (2025-11-10T09:25:25Z)
• Beyond Words: Advancing Long-Text Image Generation via Multimodal Autoregressive Models [76.68654868991517]
  Long-form text in images, such as paragraphs in slides or documents, remains a major challenge for current generative models.<n>We introduce a novel text-focused, binary tokenizer optimized for capturing detailed scene text features.<n>We develop ModelName, a multimodal autoregressive model that excels in generating high-quality long-text images with unprecedented fidelity.
  arXiv  Detail & Related papers  (2025-03-26T03:44:25Z)
• ControlText: Unlocking Controllable Fonts in Multilingual Text Rendering without Font Annotations [10.511255146429805]
  This work demonstrates that diffusion models can achieve font-controllable multilingual text rendering using just raw images without font label annotations.<n>We propose a data-driven solution that integrates the conditional diffusion model with a text segmentation model.<n>The experiment provides a proof of concept of our algorithm in zero-shot text and font editing across diverse fonts and languages.
  arXiv  Detail & Related papers  (2025-02-16T05:30:18Z)
• JoyType: A Robust Design for Multilingual Visual Text Creation [14.441897362967344]
  We introduce a novel approach for multilingual visual text creation, named JoyType.<n>JoyType is designed to maintain the font style of text during the image generation process.<n>Our evaluations, based on both visual and accuracy metrics, demonstrate that JoyType significantly outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-09-26T04:23:17Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• VQ-Font: Few-Shot Font Generation with Structure-Aware Enhancement and Quantization [52.870638830417]
  We propose a VQGAN-based framework (i.e., VQ-Font) to enhance glyph fidelity through token prior refinement and structure-aware enhancement. Specifically, we pre-train a VQGAN to encapsulate font token prior within a codebook. Subsequently, VQ-Font refines the synthesized glyphs with the codebook to eliminate the domain gap between synthesized and real-world strokes.
  arXiv  Detail & Related papers  (2023-08-27T06:32:20Z)
• CapText: Large Language Model-based Caption Generation From Image Context and Description [0.0]
  We propose and evaluate a new approach to generate captions from textual descriptions and context alone. Our approach outperforms current state-of-the-art image-text alignment models like OSCAR-VinVL on this task on the CIDEr metric.
  arXiv  Detail & Related papers  (2023-06-01T02:40:44Z)
• ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models [77.03361270726944]
  Current personalization methods can invert an object or concept into the textual conditioning space and compose new natural sentences for text-to-image diffusion models. We propose a novel approach that leverages the step-by-step generation process of diffusion models, which generate images from low to high frequency information. We apply ProSpect in various personalized attribute-aware image generation applications, such as image-guided or text-driven manipulations of materials, style, and layout.
  arXiv  Detail & Related papers  (2023-05-25T16:32:01Z)
• TextDiffuser: Diffusion Models as Text Painters [118.30923824681642]
  We introduce TextDiffuser, focusing on generating images with visually appealing text that is coherent with backgrounds. We contribute the first large-scale text images dataset with OCR annotations, MARIO-10M, containing 10 million image-text pairs. We show that TextDiffuser is flexible and controllable to create high-quality text images using text prompts alone or together with text template images, and conduct text inpainting to reconstruct incomplete images with text.
  arXiv  Detail & Related papers  (2023-05-18T10:16:19Z)
• GlyphDiffusion: Text Generation as Image Generation [100.98428068214736]
  We propose GlyphDiffusion, a novel diffusion approach for text generation via text-guided image generation. Our key idea is to render the target text as a glyph image containing visual language content. Our model also makes significant improvements compared to the recent diffusion model.
  arXiv  Detail & Related papers  (2023-04-25T02:14:44Z)
• SpaText: Spatio-Textual Representation for Controllable Image Generation [61.89548017729586]
  SpaText is a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map. We show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-conditional-based.
  arXiv  Detail & Related papers  (2022-11-25T18:59:10Z)
• DG-Font: Deformable Generative Networks for Unsupervised Font Generation [14.178381391124036]
  We propose novel deformable generative networks for unsupervised font generation (DGFont) We introduce a feature deformation skip connection (FDSC) which predicts pairs of displacement maps and employs the predicted maps to apply deformable convolution to the low-level feature maps from the content encoder. Experiments demonstrate that our model generates characters in higher quality than state-of-art methods.
  arXiv  Detail & Related papers  (2021-04-07T11:32:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.