DivCon: Divide and Conquer for Progressive Text-to-Image Generation

• URL: http://arxiv.org/abs/2403.06400v2
• Date: Fri, 16 Aug 2024 17:13:02 GMT
• Title: DivCon: Divide and Conquer for Progressive Text-to-Image Generation
• Authors: Yuhao Jia, Wenhan Tan,
• Abstract summary: Diffusion-driven text-to-image (T2I) generation has achieved remarkable advancements. layout is employed as an intermedium to bridge large language models and layout-based diffusion models. We introduce a divide-and-conquer approach which decouples the T2I generation task into simple subtasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion-driven text-to-image (T2I) generation has achieved remarkable advancements. To further improve T2I models' capability in numerical and spatial reasoning, the layout is employed as an intermedium to bridge large language models and layout-based diffusion models. However, these methods still struggle with generating images from textural prompts with multiple objects and complicated spatial relationships. To tackle this challenge, we introduce a divide-and-conquer approach which decouples the T2I generation task into simple subtasks. Our approach divides the layout prediction stage into numerical & spatial reasoning and bounding box prediction. Then, the layout-to-image generation stage is conducted in an iterative manner to reconstruct objects from easy ones to difficult ones. We conduct experiments on the HRS and NSR-1K benchmarks and our approach outperforms previous state-of-the-art models with notable margins. In addition, visual results demonstrate that our approach significantly improves the controllability and consistency in generating multiple objects from complex textural prompts.

Related papers

• A Two-Stage System for Layout-Controlled Image Generation using Large Language Models and Diffusion Models [0.0]
  Text-to-image diffusion models exhibit remarkable generative capabilities, but lack precise control over object counts and spatial arrangements.<n>This work introduces a two-stage system to address these compositional limitations.<n>The first stage employs a Large Language Model (LLM) to generate a structured layout from a list of objects.<n>The second stage uses a layout-conditioned diffusion model to synthesize a photorealistic image adhering to this layout.
  arXiv  Detail & Related papers  (2025-11-10T09:40:48Z)
• Beyond Frequency: Scoring-Driven Debiasing for Object Detection via Blueprint-Prompted Image Synthesis [97.37770785712475]
  We present a generation-based debiasing framework for object detection.<n>Our method significantly narrows the performance gap for underrepresented object groups.
  arXiv  Detail & Related papers  (2025-10-21T02:19:12Z)
• ComposeAnything: Composite Object Priors for Text-to-Image Generation [72.98469853839246]
  ComposeAnything is a novel framework for improving compositional image generation without retraining existing T2I models.<n>Our approach first leverages the chain-of-thought reasoning abilities of LLMs to produce 2.5D semantic layouts from text.<n>Our model generates high-quality images with compositions that faithfully reflect the text.
  arXiv  Detail & Related papers  (2025-05-30T00:13:36Z)
• Conditional Panoramic Image Generation via Masked Autoregressive Modeling [35.624070746282186]
  We propose a unified framework, Panoramic AutoRegressive model (PAR), which leverages masked autoregressive modeling to address these challenges.<n>To address the inherent discontinuity in existing generative models, we introduce circular padding to enhance spatial coherence.<n>Experiments demonstrate competitive performance in text-to-image generation and panorama outpainting tasks.
  arXiv  Detail & Related papers  (2025-05-22T16:20:12Z)
• CompAlign: Improving Compositional Text-to-Image Generation with a Complex Benchmark and Fine-Grained Feedback [58.27353205269664]
  State-of-the-art T2I models are capable of generating high-resolution images given textual prompts.<n>However, they struggle with accurately depicting compositional scenes that specify multiple objects, attributes, and spatial relations.<n>We present CompAlign, a challenging benchmark with an emphasis on assessing the depiction of 3D-spatial relationships.
  arXiv  Detail & Related papers  (2025-05-16T12:23:58Z)
• Aligning Text to Image in Diffusion Models is Easier Than You Think [47.623236425067326]
  We introduce a lightweight contrastive fine tuning strategy called SoftREPA that uses soft text tokens. Our method explicitly increases the mutual information between text and image representations, leading to enhanced semantic consistency.
  arXiv  Detail & Related papers  (2025-03-11T10:14:22Z)
• ToLo: A Two-Stage, Training-Free Layout-To-Image Generation Framework For High-Overlap Layouts [2.799190378263432]
  ToLo is a two-stage, training-free layout-to-image generation framework for high-overlap layouts.<n>We show that ToLo significantly enhances the performance of existing methods when dealing with high-overlap layouts.
  arXiv  Detail & Related papers  (2025-03-03T15:41:51Z)
• GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis [10.47359822447001]
  We present an alternate paradigm for T2I synthesis, decomposing the task of complex multi-step generation into three steps. Our approach derives its strength from the fact that it is modular in nature, is training free, and can be applied over any combination of image generation and editing models.
  arXiv  Detail & Related papers  (2024-12-08T22:29:56Z)
• Training-Free Layout-to-Image Generation with Marginal Attention Constraints [73.55660250459132]
  We propose a training-free layout-to-image (L2I) approach, which eliminates the need for additional modules or fine-tuning.<n>Specifically, we use text-visual cross-attention feature maps to quantify inconsistencies between the layout of the generated images and the provided instructions.<n>We leverage pixel-to-pixel correlations in the self-attention feature maps to align cross-attention maps and combine three loss functions constrained by boundary attention to update latent features.
  arXiv  Detail & Related papers  (2024-11-15T05:44:45Z)
• Generating Intermediate Representations for Compositional Text-To-Image Generation [16.757550214291015]
  We propose a compositional approach for text-to-image generation based on two stages. In the first stage, we design a diffusion-based generative model to produce one or more aligned intermediate representations conditioned on text. In the second stage, we map these representations, together with the text, to the final output image using a separate diffusion-based generative model.
  arXiv  Detail & Related papers  (2024-10-13T10:24:55Z)
• SIGMA:Sinkhorn-Guided Masked Video Modeling [69.31715194419091]
  Sinkhorn-guided Masked Video Modelling ( SIGMA) is a novel video pretraining method. We distribute features of space-time tubes evenly across a limited number of learnable clusters. Experimental results on ten datasets validate the effectiveness of SIGMA in learning more performant, temporally-aware, and robust video representations.
  arXiv  Detail & Related papers  (2024-07-22T08:04:09Z)
• Obtaining Favorable Layouts for Multiple Object Generation [50.616875565173274]
  Large-scale text-to-image models can generate high-quality and diverse images based on textual prompts. However, the existing state-of-the-art diffusion models face difficulty when generating images that involve multiple subjects. We propose a novel approach based on a guiding principle. We allow the diffusion model to initially propose a layout, and then we rearrange the layout grid. This is achieved by enforcing cross-attention maps (XAMs) to adhere to proposed masks and by migrating pixels from latent maps to new locations determined by us.
  arXiv  Detail & Related papers  (2024-05-01T18:07:48Z)
• Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion [35.21106030549071]
  Diffusion Probabilistic Models (DPMs) are dominant force in text-to-image generation tasks. We propose an alternative view of state-of-the-art DPMs as a way of inverting advanced Vision-Language Models (VLMs) By directly optimizing images with the supervision of discriminative VLMs, the proposed method can potentially achieve a better text-image alignment.
  arXiv  Detail & Related papers  (2024-02-26T05:08:40Z)
• Direct Consistency Optimization for Compositional Text-to-Image Personalization [73.94505688626651]
  Text-to-image (T2I) diffusion models, when fine-tuned on a few personal images, are able to generate visuals with a high degree of consistency. We propose to fine-tune the T2I model by maximizing consistency to reference images, while penalizing the deviation from the pretrained model.
  arXiv  Detail & Related papers  (2024-02-19T09:52:41Z)
• Skeleton-Guided Instance Separation for Fine-Grained Segmentation in Microscopy [23.848474219551818]
  One of the fundamental challenges in microscopy (MS) image analysis is instance segmentation (IS) We propose a novel one-stage framework named A2B-IS to address this challenge and enhance the accuracy of IS in MS images. Our method has been thoroughly validated on two large-scale MS datasets.
  arXiv  Detail & Related papers  (2024-01-18T11:14:32Z)
• Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive [21.49096276631859]
  Current L2I models either suffer from poor editability via text or weak alignment between the generated image and the input layout. We propose to integrate adversarial supervision into the conventional training pipeline of L2I diffusion models (ALDM) Specifically, we employ a segmentation-based discriminator which provides explicit feedback to the diffusion generator on the pixel-level alignment between the denoised image and the input layout.
  arXiv  Detail & Related papers  (2024-01-16T20:31:46Z)
• Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis [47.27044390204868]
  We introduce a novel approach to improving T2I diffusion models using Large Language Models (LLMs) as layout generators. Our experiments demonstrate significant improvements in image quality and layout accuracy.
  arXiv  Detail & Related papers  (2023-11-28T14:51:13Z)
• R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation [74.5598315066249]
  We probe into zero-shot grounded T2I generation with diffusion models. We propose a Region and Boundary (R&B) aware cross-attention guidance approach.
  arXiv  Detail & Related papers  (2023-10-13T05:48:42Z)
• SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation [68.42476385214785]
  We propose a novel Spatial-Semantic Map Guided (SSMG) diffusion model that adopts the feature map, derived from the layout, as guidance. SSMG achieves superior generation quality with sufficient spatial and semantic controllability compared to previous works. We also propose the Relation-Sensitive Attention (RSA) and Location-Sensitive Attention (LSA) mechanisms.
  arXiv  Detail & Related papers  (2023-08-20T04:09:12Z)
• LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models [62.75006608940132]
  This work proposes to enhance prompt understanding capabilities in text-to-image diffusion models. Our method leverages a pretrained large language model for grounded generation in a novel two-stage process. Our method significantly outperforms the base diffusion model and several strong baselines in accurately generating images.
  arXiv  Detail & Related papers  (2023-05-23T03:59:06Z)
• Aggregated Contextual Transformations for High-Resolution Image Inpainting [57.241749273816374]
  We propose an enhanced GAN-based model, named Aggregated COntextual-Transformation GAN (AOT-GAN) for high-resolution image inpainting. To enhance context reasoning, we construct the generator of AOT-GAN by stacking multiple layers of a proposed AOT block. For improving texture synthesis, we enhance the discriminator of AOT-GAN by training it with a tailored mask-prediction task.
  arXiv  Detail & Related papers  (2021-04-03T15:50:17Z)
• Object-Centric Image Generation from Layouts [93.10217725729468]
  We develop a layout-to-image-generation method to generate complex scenes with multiple objects. Our method learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We introduce SceneFID, an object-centric adaptation of the popular Fr'echet Inception Distance metric, that is better suited for multi-object images.
  arXiv  Detail & Related papers  (2020-03-16T21:40:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.