A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting

• URL: http://arxiv.org/abs/2312.03594v4
• Date: Tue, 23 Jul 2024 11:48:57 GMT
• Title: A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting
• Authors: Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, Kai Chen,
• Abstract summary: We introduce PowerPaint, the first high-quality and versatile inpainting model that excels in multiple inpainting tasks. We demonstrate the versatility of the task prompt in PowerPaint by showcasing its effectiveness as a negative prompt for object removal. We leverage prompt techniques to enable controllable shape-guided object inpainting, enhancing the model's applicability in shape-guided applications.
• Score: 38.53807472111521
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Advancing image inpainting is challenging as it requires filling user-specified regions for various intents, such as background filling and object synthesis. Existing approaches focus on either context-aware filling or object synthesis using text descriptions. However, achieving both tasks simultaneously is challenging due to differing training strategies. To overcome this challenge, we introduce PowerPaint, the first high-quality and versatile inpainting model that excels in multiple inpainting tasks. First, we introduce learnable task prompts along with tailored fine-tuning strategies to guide the model's focus on different inpainting targets explicitly. This enables PowerPaint to accomplish various inpainting tasks by utilizing different task prompts, resulting in state-of-the-art performance. Second, we demonstrate the versatility of the task prompt in PowerPaint by showcasing its effectiveness as a negative prompt for object removal. Moreover, we leverage prompt interpolation techniques to enable controllable shape-guided object inpainting, enhancing the model's applicability in shape-guided applications. Finally, we conduct extensive experiments and applications to verify the effectiveness of PowerPaint. We release our codes and models on our project page: https://powerpaint.github.io/.

Related papers

• VIP: Versatile Image Outpainting Empowered by Multimodal Large Language Model [76.02314305164595]
  This work presents a novel image outpainting framework that is capable of customizing the results according to the requirement of users. We take advantage of a Multimodal Large Language Model (MLLM) that automatically extracts and organizes the corresponding textual descriptions of the masked and unmasked part of a given image. In addition, a special Cross-Attention module, namely Center-Total-Surrounding (CTS), is elaborately designed to enhance further the the interaction between specific space regions of the image and corresponding parts of the text prompts.
  arXiv  Detail & Related papers  (2024-06-03T07:14:19Z)
• MOWA: Multiple-in-One Image Warping Model [65.73060159073644]
  We propose a Multiple-in-One image warping model (named MOWA) in this work. We mitigate the difficulty of multi-task learning by disentangling the motion estimation at both the region level and pixel level. To our knowledge, this is the first work that solves multiple practical warping tasks in one single model.
  arXiv  Detail & Related papers  (2024-04-16T16:50:35Z)
• HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models [59.01600111737628]
  HD-Painter is a training free approach that accurately follows prompts and coherently scales to high resolution image inpainting. To this end, we design the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention scores. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-12-21T18:09:30Z)
• Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model [19.800236358666123]
  We propose Uni-paint, a unified framework for multimodal inpainting. Uni-paint offers various modes of guidance, including text-driven, stroke-driven, exemplar-driven inpainting. Our approach achieves comparable results to existing single-modal methods.
  arXiv  Detail & Related papers  (2023-10-11T06:11:42Z)
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions [12.792576041526287]
  PromptPaint allows users to mix prompts that express challenging concepts. We characterize different approaches for mixing prompts, design trade-offs, and socio-technical challenges for generative models.
  arXiv  Detail & Related papers  (2023-08-09T18:41:11Z)
• SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model [27.91089554671927]
  Generic image inpainting aims to complete a corrupted image by borrowing surrounding information. By contrast, multi-modal inpainting provides more flexible and useful controls on the inpainted content. We propose a new diffusion-based model named SmartBrush for completing a missing region with an object using both text and shape-guidance.
  arXiv  Detail & Related papers  (2022-12-09T18:36:13Z)
• Images Speak in Images: A Generalist Painter for In-Context Visual Learning [98.78475432114595]
  In-context learning allows the model to rapidly adapt to various tasks with only a handful of prompts and examples. It is unclear how to define the general-purpose task prompts that the vision model can understand and transfer to out-of-domain tasks. We present Painter, a generalist model which redefines the output of core vision tasks as images, and specify task prompts as also images.
  arXiv  Detail & Related papers  (2022-12-05T18:59:50Z)
• Learning Prior Feature and Attention Enhanced Image Inpainting [63.21231753407192]
  This paper incorporates the pre-training based Masked AutoEncoder (MAE) into the inpainting model. We propose to use attention priors from MAE to make the inpainting model learn more long-distance dependencies between masked and unmasked regions.
  arXiv  Detail & Related papers  (2022-08-03T04:32:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.