AID: Attention Interpolation of Text-to-Image Diffusion

• URL: http://arxiv.org/abs/2403.17924v3
• Date: Fri, 04 Oct 2024 17:09:40 GMT
• Title: AID: Attention Interpolation of Text-to-Image Diffusion
• Authors: Qiyuan He, Jinghao Wang, Ziwei Liu, Angela Yao,
• Abstract summary: We introduce a training-free technique named Attention Interpolation via Diffusion (AID) AID fuses the interpolated attention with self-attention to boost fidelity. We also present a variant, Conditional-guided Attention Interpolation via Diffusion (AID), that considers as a condition-dependent generative process.
• Score: 64.87754163416241
• License:
• Abstract: Conditional diffusion models can create unseen images in various settings, aiding image interpolation. Interpolation in latent spaces is well-studied, but interpolation with specific conditions like text or poses is less understood. Simple approaches, such as linear interpolation in the space of conditions, often result in images that lack consistency, smoothness, and fidelity. To that end, we introduce a novel training-free technique named Attention Interpolation via Diffusion (AID). Our key contributions include 1) proposing an inner/outer interpolated attention layer; 2) fusing the interpolated attention with self-attention to boost fidelity; and 3) applying beta distribution to selection to increase smoothness. We also present a variant, Prompt-guided Attention Interpolation via Diffusion (PAID), that considers interpolation as a condition-dependent generative process. This method enables the creation of new images with greater consistency, smoothness, and efficiency, and offers control over the exact path of interpolation. Our approach demonstrates effectiveness for conceptual and spatial interpolation. Code and demo are available at https://github.com/QY-H00/attention-interpolation-diffusion.

Related papers

• DreamMover: Leveraging the Prior of Diffusion Models for Image Interpolation with Large Motion [35.60459492849359]
  We study the problem of generating intermediate images from image pairs with large motion. Due to the large motion, the intermediate semantic information may be absent in input images. We propose DreamMover, a novel image framework with three main components.
  arXiv  Detail & Related papers  (2024-09-15T04:09:12Z)
• DiffMorpher: Unleashing the Capability of Diffusion Models for Image Morphing [28.593023489682654]
  We present DiffMorpher, the first approach enabling smooth and natural image morphing using diffusion models. Our key idea is to capture the semantics of the two images by fitting two LoRAs to them respectively, and interpolate between both the LoRA parameters and the latent noises to ensure a smooth semantic transition. In addition, we propose an attention and injection technique and a new sampling schedule to further enhance the smoothness between consecutive images.
  arXiv  Detail & Related papers  (2023-12-12T16:28:08Z)
• IMPUS: Image Morphing with Perceptually-Uniform Sampling Using Diffusion Models [24.382275473592046]
  We present a diffusion-based image morphing approach with perceptually-uniform sampling (IMPUS) IMPUS produces smooth, direct and realistic adaptations given an image pair.
  arXiv  Detail & Related papers  (2023-11-12T10:03:32Z)
• Mutual-Guided Dynamic Network for Image Fusion [51.615598671899335]
  We propose a novel mutual-guided dynamic network (MGDN) for image fusion, which allows for effective information utilization across different locations and inputs. Experimental results on five benchmark datasets demonstrate that our proposed method outperforms existing methods on four image fusion tasks.
  arXiv  Detail & Related papers  (2023-08-24T03:50:37Z)
• Interpolating between Images with Diffusion Models [2.6027967363792865]
  Interpolating between two input images is a task missing from image generation pipelines. We propose a method for zero-shot using latent diffusion models. For greater consistency, or to specify additional criteria, we can generate several candidates and use CLIP to select the highest quality image.
  arXiv  Detail & Related papers  (2023-07-24T07:03:22Z)
• Harnessing the Spatial-Temporal Attention of Diffusion Models for High-Fidelity Text-to-Image Synthesis [59.10787643285506]
  Diffusion-based models have achieved state-of-the-art performance on text-to-image synthesis tasks. One critical limitation of these models is the low fidelity of generated images with respect to the text description. We propose a new text-to-image algorithm that adds explicit control over spatial-temporal cross-attention in diffusion models.
  arXiv  Detail & Related papers  (2023-04-07T23:49:34Z)
• Smooth image-to-image translations with latent space interpolations [64.8170758294427]
  Multi-domain image-to-image (I2I) translations can transform a source image according to the style of a target domain. We show that our regularization techniques can improve the state-of-the-art I2I translations by a large margin.
  arXiv  Detail & Related papers  (2022-10-03T11:57:30Z)
• Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label Diffusion [51.11295961195151]
  We exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels. Based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion scheme. Our scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets.
  arXiv  Detail & Related papers  (2022-06-10T05:16:50Z)
• Smoothing the Disentangled Latent Style Space for Unsupervised Image-to-Image Translation [56.55178339375146]
  Image-to-Image (I2I) multi-domain translation models are usually evaluated also using the quality of their semantic results. We propose a new training protocol based on three specific losses which help a translation network to learn a smooth and disentangled latent style space.
  arXiv  Detail & Related papers  (2021-06-16T17:58:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.