SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation

• URL: http://arxiv.org/abs/2501.03490v1
• Date: Tue, 07 Jan 2025 03:18:15 GMT
• Title: SceneBooth: Diffusion-based Framework for Subject-preserved Text-to-Image Generation
• Authors: Shang Chai, Zihang Lin, Min Zhou, Xubin Li, Liansheng Zhuang, Houqiang Li,
• Abstract summary: Existing methods often learn subject representation and incorporate it into the prompt embedding to guide image generation.<n>This paper approaches a novel framework named SceneBooth for subject-preserved text-to-image generation.<n>Our SceneBooth fixes the given subject image and generates its background image guided by the text prompts.
• Score: 46.43776651071455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to the demand for personalizing image generation, subject-driven text-to-image generation method, which creates novel renditions of an input subject based on text prompts, has received growing research interest. Existing methods often learn subject representation and incorporate it into the prompt embedding to guide image generation, but they struggle with preserving subject fidelity. To solve this issue, this paper approaches a novel framework named SceneBooth for subject-preserved text-to-image generation, which consumes inputs of a subject image, object phrases and text prompts. Instead of learning the subject representation and generating a subject, our SceneBooth fixes the given subject image and generates its background image guided by the text prompts. To this end, our SceneBooth introduces two key components, i.e., a multimodal layout generation module and a background painting module. The former determines the position and scale of the subject by generating appropriate scene layouts that align with text captions, object phrases, and subject visual information. The latter integrates two adapters (ControlNet and Gated Self-Attention) into the latent diffusion model to generate a background that harmonizes with the subject guided by scene layouts and text descriptions. In this manner, our SceneBooth ensures accurate preservation of the subject's appearance in the output. Quantitative and qualitative experimental results demonstrate that SceneBooth significantly outperforms baseline methods in terms of subject preservation, image harmonization and overall quality.

Related papers

• DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models [115.62816053600085]
  We present DesignDiffusion, a framework for synthesizing design images from textual descriptions. The proposed framework directly synthesizes textual and visual design elements from user prompts. It utilizes a distinctive character embedding derived from the visual text to enhance the input prompt.
  arXiv  Detail & Related papers  (2025-03-03T15:22:57Z)
• Bringing Characters to New Stories: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting [71.29100512700064]
  We present T-Prompter, a training-free method for theme-specific image generation. T-Prompter integrates reference images into generative models, allowing users to seamlessly specify the target theme. Our approach enables consistent story generation, character design, realistic character generation, and style-guided image generation.
  arXiv  Detail & Related papers  (2025-01-26T19:01:19Z)
• Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot Subject-Driven Image Generator [44.620847608977776]
  Diptych Prompting is a novel zero-shot approach that reinterprets as an inpainting task with precise subject alignment. Our method supports not only subject-driven generation but also stylized image generation and subject-driven image editing.
  arXiv  Detail & Related papers  (2024-11-23T06:17:43Z)
• Visual Text Generation in the Wild [67.37458807253064]
  We propose a visual text generator (termed SceneVTG) which can produce high-quality text images in the wild. The proposed SceneVTG significantly outperforms traditional rendering-based methods and recent diffusion-based methods in terms of fidelity and reasonability. The generated images provide superior utility for tasks involving text detection and text recognition.
  arXiv  Detail & Related papers  (2024-07-19T09:08:20Z)
• Locate, Assign, Refine: Taming Customized Image Inpainting with Text-Subject Guidance [17.251982243534144]
  LAR-Gen is a novel approach for image inpainting that enables seamless inpainting of masked scene images. Our approach adopts a coarse-to-fine manner to ensure subject identity preservation and local semantic coherence. Experiments and varied application scenarios demonstrate the superiority of LAR-Gen in terms of both identity preservation and text semantic consistency.
  arXiv  Detail & Related papers  (2024-03-28T16:07:55Z)
• Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model [31.819060415422353]
  Diff-Text is a training-free scene text generation framework for any language. Our method outperforms the existing method in both the accuracy of text recognition and the naturalness of foreground-background blending.
  arXiv  Detail & Related papers  (2023-12-19T15:18:40Z)
• DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Models [37.133727797607676]
  This study introduces Text-Guided Subject-Driven Image Inpainting. We compute dense subject features to ensure accurate subject replication. We employ a discriminative token selection module to eliminate redundant subject details.
  arXiv  Detail & Related papers  (2023-12-05T22:23:19Z)
• InstructBooth: Instruction-following Personalized Text-to-Image Generation [30.89054609185801]
  InstructBooth is a novel method designed to enhance image-text alignment in personalized text-to-image models. Our approach first personalizes text-to-image models with a small number of subject-specific images using a unique identifier. After personalization, we fine-tune personalized text-to-image models using reinforcement learning to maximize a reward that quantifies image-text alignment.
  arXiv  Detail & Related papers  (2023-12-04T20:34:46Z)
• LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts [60.54912319612113]
  Diffusion-based generative models have significantly advanced text-to-image generation but encounter challenges when processing lengthy and intricate text prompts. We present a novel approach leveraging Large Language Models (LLMs) to extract critical components from text prompts. Our evaluation on complex prompts featuring multiple objects demonstrates a substantial improvement in recall compared to baseline diffusion models.
  arXiv  Detail & Related papers  (2023-10-16T17:57:37Z)
• LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation [121.45667242282721]
  We propose a coarse-to-fine paradigm to achieve layout planning and image generation. Our proposed method outperforms the state-of-the-art models in terms of photorealistic layout and image generation.
  arXiv  Detail & Related papers  (2023-08-09T17:45:04Z)
• Paste, Inpaint and Harmonize via Denoising: Subject-Driven Image Editing with Pre-Trained Diffusion Model [22.975965453227477]
  We introduce a new framework called textitPaste, Inpaint and Harmonize via Denoising (PhD) In our experiments, we apply PhD to both subject-driven image editing tasks and explore text-driven scene generation given a reference subject.
  arXiv  Detail & Related papers  (2023-06-13T07:43:10Z)
• DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation [50.39533637201273]
  We propose DisenBooth, an identity-preserving disentangled tuning framework for subject-driven text-to-image generation. By combining the identity-preserved embedding and identity-irrelevant embedding, DisenBooth demonstrates more generation flexibility and controllability.
  arXiv  Detail & Related papers  (2023-05-05T09:08:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.