Text-to-Image Synthesis: A Decade Survey

• URL: http://arxiv.org/abs/2411.16164v1
• Date: Mon, 25 Nov 2024 07:40:32 GMT
• Title: Text-to-Image Synthesis: A Decade Survey
• Authors: Nonghai Zhang, Hao Tang,
• Abstract summary: Text-to-image synthesis (T2I) focuses on generating high-quality images from textual descriptions. In this survey, we review over 440 recent works on T2I.
• Score: 7.250878248686215
• License:
• Abstract: When humans read a specific text, they often visualize the corresponding images, and we hope that computers can do the same. Text-to-image synthesis (T2I), which focuses on generating high-quality images from textual descriptions, has become a significant aspect of Artificial Intelligence Generated Content (AIGC) and a transformative direction in artificial intelligence research. Foundation models play a crucial role in T2I. In this survey, we review over 440 recent works on T2I. We start by briefly introducing how GANs, autoregressive models, and diffusion models have been used for image generation. Building on this foundation, we discuss the development of these models for T2I, focusing on their generative capabilities and diversity when conditioned on text. We also explore cutting-edge research on various aspects of T2I, including performance, controllability, personalized generation, safety concerns, and consistency in content and spatial relationships. Furthermore, we summarize the datasets and evaluation metrics commonly used in T2I research. Finally, we discuss the potential applications of T2I within AIGC, along with the challenges and future research opportunities in this field.

Related papers

• Evaluating the Generation of Spatial Relations in Text and Image Generative Models [4.281091463408283]
  spatial relations are naturally understood in a visuo-spatial manner. We develop an approach to convert LLM outputs into an image, thereby allowing us to evaluate both T2I models and LLMs. Surprisingly, we found that T2I models only achieve subpar performance despite their impressive general image-generation abilities.
  arXiv  Detail & Related papers  (2024-11-12T09:30:02Z)
• PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models [50.33699462106502]
  Text-to-image (T2I) models frequently fail to produce images consistent with physical commonsense. Current T2I evaluation benchmarks focus on metrics such as accuracy, bias, and safety, neglecting the evaluation of models' internal knowledge. We introduce PhyBench, a comprehensive T2I evaluation dataset comprising 700 prompts across 4 primary categories: mechanics, optics, thermodynamics, and material properties.
  arXiv  Detail & Related papers  (2024-06-17T17:49:01Z)
• Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense? [97.0899853256201]
  We present a novel task and benchmark for evaluating the ability of text-to-image generation models to produce images that align with commonsense in real life. We evaluate whether T2I models can conduct visual-commonsense reasoning, e.g. produce images that fit "the lightbulb is unlit" vs. "the lightbulb is lit" We benchmark a variety of state-of-the-art (sota) T2I models and surprisingly find that, there is still a large gap between image synthesis and real life photos.
  arXiv  Detail & Related papers  (2024-06-11T17:59:48Z)
• ANCHOR: LLM-driven News Subject Conditioning for Text-to-Image Synthesis [6.066100464517522]
  We introduce the Abstractive News Captions with High-level cOntext Representation dataset, containing 70K+ samples sourced from 5 different news media organizations. Our proposed method Subject-Aware Finetuning (SAFE), selects and enhances the representation of key subjects in synthesized images by leveraging LLM-generated subject weights. It also adapts to the domain distribution of news images and captions through custom Domain Fine-tuning, outperforming current T2I baselines on ANCHOR.
  arXiv  Detail & Related papers  (2024-04-15T21:19:10Z)
• SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data [73.23388142296535]
  SELMA improves the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets. We show that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks. We also show that fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data.
  arXiv  Detail & Related papers  (2024-03-11T17:35:33Z)
• A Contrastive Compositional Benchmark for Text-to-Image Synthesis: A Study with Unified Text-to-Image Fidelity Metrics [58.83242220266935]
  We introduce Winoground-T2I, a benchmark designed to evaluate the compositionality of T2I models. This benchmark includes 11K complex, high-quality contrastive sentence pairs spanning 20 categories. We use Winoground-T2I with a dual objective: to evaluate the performance of T2I models and the metrics used for their evaluation.
  arXiv  Detail & Related papers  (2023-12-04T20:47:48Z)
• Mini-DALLE3: Interactive Text to Image by Prompting Large Language Models [71.49054220807983]
  A prevalent limitation persists in the effective communication with T2I models, such as Stable Diffusion, using natural language descriptions. Inspired by the recently released DALLE3, we revisit the existing T2I systems endeavoring to align human intent and introduce a new task - interactive text to image (iT2I) We present a simple approach that augments LLMs for iT2I with prompting techniques and off-the-shelf T2I models.
  arXiv  Detail & Related papers  (2023-10-11T16:53:40Z)
• Benchmarking Spatial Relationships in Text-to-Image Generation [102.62422723894232]
  We investigate the ability of text-to-image models to generate correct spatial relationships among objects. We present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image. Our experiments reveal a surprising finding that, although state-of-the-art T2I models exhibit high image quality, they are severely limited in their ability to generate multiple objects or the specified spatial relations between them.
  arXiv  Detail & Related papers  (2022-12-20T06:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.