SLayR: Scene Layout Generation with Rectified Flow

• URL: http://arxiv.org/abs/2412.05003v1
• Date: Fri, 06 Dec 2024 12:58:58 GMT
• Title: SLayR: Scene Layout Generation with Rectified Flow
• Authors: Cameron Braunstein, Hevra Petekkaya, Jan Eric Lenssen, Mariya Toneva, Eddy Ilg,
• Abstract summary: SLayR presents a novel transformer-based rectified flow model for layout generation over a token space. We show that established metrics for generated images are inconclusive for evaluating their underlying scene layout. In contrast to previous works, which perform well in either high variety or plausibility, we show that our approach performs well on both of these axes at the same time.
• Score: 10.449737374910619
• License:
• Abstract: We introduce SLayR, Scene Layout Generation with Rectified flow. State-of-the-art text-to-image models achieve impressive results. However, they generate images end-to-end, exposing no fine-grained control over the process. SLayR presents a novel transformer-based rectified flow model for layout generation over a token space that can be decoded into bounding boxes and corresponding labels, which can then be transformed into images using existing models. We show that established metrics for generated images are inconclusive for evaluating their underlying scene layout, and introduce a new benchmark suite, including a carefully designed repeatable human-evaluation procedure that assesses the plausibility and variety of generated layouts. In contrast to previous works, which perform well in either high variety or plausibility, we show that our approach performs well on both of these axes at the same time. It is also at least 5x times smaller in the number of parameters and 37% faster than the baselines. Our complete text-to-image pipeline demonstrates the added benefits of an interpretable and editable intermediate representation.

Related papers

• DivCon: Divide and Conquer for Progressive Text-to-Image Generation [0.0]
  Diffusion-driven text-to-image (T2I) generation has achieved remarkable advancements. layout is employed as an intermedium to bridge large language models and layout-based diffusion models. We introduce a divide-and-conquer approach which decouples the T2I generation task into simple subtasks.
  arXiv  Detail & Related papers  (2024-03-11T03:24:44Z)
• LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [121.45667242282721]
  We propose a coarse-to-fine paradigm to achieve layout planning and image generation. Our proposed method outperforms the state-of-the-art models in terms of photorealistic layout and image generation.
  arXiv  Detail & Related papers  (2023-08-09T17:45:04Z)
• Zero-shot spatial layout conditioning for text-to-image diffusion models [52.24744018240424]
  Large-scale text-to-image diffusion models have significantly improved the state of the art in generative image modelling. We consider image generation from text associated with segments on the image canvas, which combines an intuitive natural language interface with precise spatial control over the generated content. We propose ZestGuide, a zero-shot segmentation guidance approach that can be plugged into pre-trained text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-06-23T19:24:48Z)
• Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation [147.81509219686419]
  We propose a diagnostic benchmark for layout-guided image generation that examines four categories of spatial control skills: number, position, size, and shape. Next, we propose IterInpaint, a new baseline that generates foreground and background regions step-by-step via inpainting. We show comprehensive ablation studies on IterInpaint, including training task ratio, crop&paste vs. repaint, and generation order.
  arXiv  Detail & Related papers  (2023-04-13T16:58:33Z)
• LayoutDiffuse: Adapting Foundational Diffusion Models for Layout-to-Image Generation [24.694298869398033]
  Our method trains efficiently, generates images with both high perceptual quality and layout alignment. Our method significantly outperforms other 10 generative models based on GANs, VQ-VAE, and diffusion models.
  arXiv  Detail & Related papers  (2023-02-16T14:20:25Z)
• SpaText: Spatio-Textual Representation for Controllable Image Generation [61.89548017729586]
  SpaText is a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map. We show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-conditional-based.
  arXiv  Detail & Related papers  (2022-11-25T18:59:10Z)
• SceneComposer: Any-Level Semantic Image Synthesis [80.55876413285587]
  We propose a new framework for conditional image synthesis from semantic layouts of any precision levels. The framework naturally reduces to text-to-image (T2I) at the lowest level with no shape information, and it becomes segmentation-to-image (S2I) at the highest level. We introduce several novel techniques to address the challenges coming with this new setup.
  arXiv  Detail & Related papers  (2022-11-21T18:59:05Z)
• eDiffi: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers [87.52504764677226]
  Large-scale diffusion-based generative models have led to breakthroughs in text-conditioned high-resolution image synthesis. We train an ensemble of text-to-image diffusion models specialized for different stages synthesis. Our ensemble of diffusion models, called eDiffi, results in improved text alignment while maintaining the same inference cost.
  arXiv  Detail & Related papers  (2022-11-02T17:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.