IterInv: Iterative Inversion for Pixel-Level T2I Models

• URL: http://arxiv.org/abs/2310.19540v2
• Date: Mon, 22 Apr 2024 02:03:02 GMT
• Title: IterInv: Iterative Inversion for Pixel-Level T2I Models
• Authors: Chuanming Tang, Kai Wang, Joost van de Weijer,
• Abstract summary: DDIM inversion is a prevalent practice rooted in Latent Diffusion Models (LDM) Large pretrained T2I models working on the latent space suffer from losing details due to the first compression stage with an autoencoder mechanism. We develop an iterative inversion (IterInv) technique for this category of T2I models and verify IterInv with the open-source DeepFloyd-IF model.
• Score: 16.230193725587807
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale text-to-image diffusion models have been a ground-breaking development in generating convincing images following an input text prompt. The goal of image editing research is to give users control over the generated images by modifying the text prompt. Current image editing techniques predominantly hinge on DDIM inversion as a prevalent practice rooted in Latent Diffusion Models (LDM). However, the large pretrained T2I models working on the latent space suffer from losing details due to the first compression stage with an autoencoder mechanism. Instead, other mainstream T2I pipeline working on the pixel level, such as Imagen and DeepFloyd-IF, circumvents the above problem. They are commonly composed of multiple stages, typically starting with a text-to-image stage and followed by several super-resolution stages. In this pipeline, the DDIM inversion fails to find the initial noise and generate the original image given that the super-resolution diffusion models are not compatible with the DDIM technique. According to our experimental findings, iteratively concatenating the noisy image as the condition is the root of this problem. Based on this observation, we develop an iterative inversion (IterInv) technique for this category of T2I models and verify IterInv with the open-source DeepFloyd-IF model.Specifically, IterInv employ NTI as the inversion and reconstruction of low-resolution image generation. In stages 2 and 3, we update the latent variance at each timestep to find the deterministic inversion trace and promote the reconstruction process. By combining our method with a popular image editing method, we prove the application prospects of IterInv. The code will be released upon acceptance. The code is available at \url{https://github.com/Tchuanm/IterInv.git}.

Related papers

• DeepInv: A Novel Self-supervised Learning Approach for Fast and Accurate Diffusion Inversion [65.5172878666262]
  Diffusion inversion is a challenging task due to the lack of viable supervision signals.<n>We propose a novel self-supervised diffusion inversion approach, termed Deep Inversion (DeepInv)<n>DeepInv is also equipped with an iterative and multi-scale training regime to train a parameterized inversion solver.
  arXiv  Detail & Related papers  (2026-01-04T11:27:26Z)
• Qwen-Image-Layered: Towards Inherent Editability via Layer Decomposition [73.43121650616804]
  We propose textbfQwen-Image-Layered, an end-to-end diffusion model that decomposes a single RGB image into multiple semantically disentangled RGBA layers.<n>Our method significantly surpasses existing approaches in decomposition quality and establishes a new paradigm for consistent image editing.
  arXiv  Detail & Related papers  (2025-12-17T17:12:42Z)
• Test-time Prompt Refinement for Text-to-Image Models [14.505841027491114]
  We introduce a test-time prompt refinement framework that requires no additional training of the underlying T2I model, termed TIR.<n>In our approach, each generation step is followed by a refinement step, where a pretrained multimodal large language model (MLLM) analyzes the output image and the user's prompt.<n>We demonstrate that this closed-loop strategy improves alignment and visual coherence across multiple benchmark datasets, all while maintaining plug-and-play integration with black-box T2I models.
  arXiv  Detail & Related papers  (2025-07-22T20:30:13Z)
• MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidance [11.023241681116295]
  We propose a unified framework for multi-task, multi-degradation, and language-guided image fusion. Our framework includes two key components: 1) a practical degradation pipeline that simulates real-world image degradations and generates interactive prompts to guide the model; 2) an all-in-one Transformer (DiT) operating in latent space, which fuses a clean image conditioned on both the degraded inputs and the generated prompts.
  arXiv  Detail & Related papers  (2025-03-19T07:20:02Z)
• Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model [31.70050311326183]
  Diffusion models tend to generate videos with less motion than expected. We address this issue from both inference and training aspects. Our methods outperform baselines by producing higher motion scores with lower errors.
  arXiv  Detail & Related papers  (2024-06-22T04:56:16Z)
• Blind Image Restoration via Fast Diffusion Inversion [17.139433082780037]
  Blind Image Restoration via fast Diffusion (BIRD) is a blind IR method that jointly optimize for the degradation model parameters and the restored image. A key idea in our method is not to modify the reverse sampling, i.e., not to alter all the intermediate latents, once an initial noise is sampled. We experimentally validate BIRD on several image restoration tasks and show that it achieves state of the art performance on all of them.
  arXiv  Detail & Related papers  (2024-05-29T23:38:12Z)
• DivCon: Divide and Conquer for Progressive Text-to-Image Generation [0.0]
  Diffusion-driven text-to-image (T2I) generation has achieved remarkable advancements. layout is employed as an intermedium to bridge large language models and layout-based diffusion models. We introduce a divide-and-conquer approach which decouples the T2I generation task into simple subtasks.
  arXiv  Detail & Related papers  (2024-03-11T03:24:44Z)
• Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models [94.25020178662392]
  Text-to-image (T2I) research has grown explosively in the past year. One pain point persists: the text prompt engineering, and searching high-quality text prompts for customized results is more art than science. In this paper, we take "Text" out of a pre-trained T2I diffusion model, to reduce the burdensome prompt engineering efforts for users.
  arXiv  Detail & Related papers  (2023-05-25T16:30:07Z)
• BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing [73.74570290836152]
  BLIP-Diffusion is a new subject-driven image generation model that supports multimodal control. Unlike other subject-driven generation models, BLIP-Diffusion introduces a new multimodal encoder which is pre-trained to provide subject representation.
  arXiv  Detail & Related papers  (2023-05-24T04:51:04Z)
• If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selection [53.320946030761796]
  diffusion-based text-to-image (T2I) models can lack faithfulness to the text prompt. We show that large T2I diffusion models are more faithful than usually assumed, and can generate images faithful to even complex prompts. We introduce a pipeline that generates candidate images for a text prompt and picks the best one according to an automatic scoring system.
  arXiv  Detail & Related papers  (2023-05-22T17:59:41Z)
• Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization [73.52943587514386]
  Existing vector quantization (VQ) based autoregressive models follow a two-stage generation paradigm. We propose a novel two-stage framework: (1) Dynamic-Quantization VAE (DQ-VAE) which encodes image regions into variable-length codes based their information densities for accurate representation.
  arXiv  Detail & Related papers  (2023-05-19T14:56:05Z)
• Training-free Content Injection using h-space in Diffusion Models [16.51521884698886]
  In this paper, we introduce a method to inject the content of one image into another image by combining their features in the generative processes. Unlike custom-diffusion approaches, our method does not require time-consuming optimization or fine-tuning.
  arXiv  Detail & Related papers  (2023-03-27T17:19:50Z)
• Restormer: Efficient Transformer for High-Resolution Image Restoration [118.9617735769827]
  convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data. Transformers have shown significant performance gains on natural language and high-level vision tasks. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks.
  arXiv  Detail & Related papers  (2021-11-18T18:59:10Z)
• Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network [73.5062435623908]
  We propose a new I2I translation method that generates a new model in the target domain via a series of model transformations. By feeding the latent vector into the generated model, we can perform I2I translation between the source domain and target domain.
  arXiv  Detail & Related papers  (2020-10-12T13:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.