TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting

• URL: http://arxiv.org/abs/2504.00996v1
• Date: Tue, 01 Apr 2025 17:33:29 GMT
• Title: TurboFill: Adapting Few-step Text-to-image Model for Fast Image Inpainting
• Authors: Liangbin Xie, Daniil Pakhomov, Zhonghao Wang, Zongze Wu, Ziyan Chen, Yuqian Zhou, Haitian Zheng, Zhifei Zhang, Zhe Lin, Jiantao Zhou, Chao Dong,
• Abstract summary: TurboFill is a fast image inpainting model that enhances a few-step text-to-image diffusion model with an inpainting adapter for high-quality and efficient inpainting.<n>We overcome this by training an inpainting adapter on a few-step distilled text-to-image model, DMD2, using a novel 3-step adversarial training scheme.<n>Experiments show that TurboFill outperforms both multi-step BrushNet and few-step inpainting methods, setting a new benchmark for high-performance inpainting tasks.
• Score: 51.015989674456364
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces TurboFill, a fast image inpainting model that enhances a few-step text-to-image diffusion model with an inpainting adapter for high-quality and efficient inpainting. While standard diffusion models generate high-quality results, they incur high computational costs. We overcome this by training an inpainting adapter on a few-step distilled text-to-image model, DMD2, using a novel 3-step adversarial training scheme to ensure realistic, structurally consistent, and visually harmonious inpainted regions. To evaluate TurboFill, we propose two benchmarks: DilationBench, which tests performance across mask sizes, and HumanBench, based on human feedback for complex prompts. Experiments show that TurboFill outperforms both multi-step BrushNet and few-step inpainting methods, setting a new benchmark for high-performance inpainting tasks. Our project page: https://liangbinxie.github.io/projects/TurboFill/

Related papers

• Show-o Turbo: Towards Accelerated Unified Multimodal Understanding and Generation [12.34529497235534]
  Show-o is a multimodal understanding model for text-to-image and image-to-text generation. This paper introduces Show-o Turbo to bridge the gap between Show-o and other approaches. Show-o Turbo exhibits a 1.5x speedup without significantly sacrificing performance.
  arXiv  Detail & Related papers  (2025-02-08T02:52:25Z)
• Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Framework [3.7953598825170753]
  Kandinsky 3 is a novel T2I model based on latent diffusion, achieving a high level of quality and photorealism. We extend the base T2I model for various applications and create a multifunctional generation system. Human evaluations show that Kandinsky 3 demonstrates one of the highest quality scores among open source generation systems.
  arXiv  Detail & Related papers  (2024-10-28T14:22:08Z)
• FlowTurbo: Towards Real-time Flow-Based Image Generation with Velocity Refiner [70.90505084288057]
  Flow-based models tend to produce a straighter sampling trajectory during the sampling process. We introduce several techniques including a pseudo corrector and sample-aware compilation to further reduce inference time. FlowTurbo reaches an FID of 2.12 on ImageNet with 100 (ms / img) and FID of 3.93 with 38 (ms / img)
  arXiv  Detail & Related papers  (2024-09-26T17:59:51Z)
• BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion [61.90969199199739]
  BrushNet is a novel plug-and-play dual-branch model engineered to embed pixel-level masked image features into any pre-trained DM. BrushNet's superior performance over existing models across seven key metrics, including image quality, mask region preservation, and textual coherence.
  arXiv  Detail & Related papers  (2024-03-11T17:59:31Z)
• Repaint123: Fast and High-quality One Image to 3D Generation with Progressive Controllable 2D Repainting [16.957766297050707]
  We present Repaint123 to alleviate multi-view bias as well as texture degradation and speed up the generation process. We propose visibility-aware adaptive repainting strength for overlap regions to enhance the generated image quality. Our method has a superior ability to generate high-quality 3D content with multi-view consistency and fine textures in 2 minutes from scratch.
  arXiv  Detail & Related papers  (2023-12-20T18:51:02Z)
• SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation [1.5892730797514436]
  Text-to-image diffusion models often suffer from slow iterative sampling processes. We present a novel image-free distillation scheme named $textbfSwiftBrush$. SwiftBrush achieves an FID score of $textbf16.67$ and a CLIP score of $textbf0.29$ on the COCO-30K benchmark.
  arXiv  Detail & Related papers  (2023-12-08T18:44:09Z)
• Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model [68.98311213582949]
  We propose Instant3D, a novel method that generates high-quality and diverse 3D assets from text prompts in a feed-forward manner. Our method can generate diverse 3D assets of high visual quality within 20 seconds, two orders of magnitude faster than previous optimization-based methods.
  arXiv  Detail & Related papers  (2023-11-10T18:03:44Z)
• SnapFusion: Text-to-Image Diffusion Model on Mobile Devices within Two Seconds [88.06788636008051]
  Text-to-image diffusion models can create stunning images from natural language descriptions that rival the work of professional artists and photographers. These models are large, with complex network architectures and tens of denoising iterations, making them computationally expensive and slow to run. We present a generic approach that unlocks running text-to-image diffusion models on mobile devices in less than $2$ seconds.
  arXiv  Detail & Related papers  (2023-06-01T17:59:25Z)
• T-former: An Efficient Transformer for Image Inpainting [50.43302925662507]
  A class of attention-based network architectures, called transformer, has shown significant performance on natural language processing fields. In this paper, we design a novel attention linearly related to the resolution according to Taylor expansion, and based on this attention, a network called $T$-former is designed for image inpainting. Experiments on several benchmark datasets demonstrate that our proposed method achieves state-of-the-art accuracy while maintaining a relatively low number of parameters and computational complexity.
  arXiv  Detail & Related papers  (2023-05-12T04:10:42Z)
• Learning Sparse Masks for Diffusion-based Image Inpainting [10.633099921979674]
  Diffusion-based inpainting is a powerful tool for the reconstruction of images from sparse data. We provide a model for highly efficient adaptive mask generation. Experiments indicate that our model can achieve competitive quality with an acceleration by as much as four orders of magnitude.
  arXiv  Detail & Related papers  (2021-10-06T10:20:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.