Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models

• URL: http://arxiv.org/abs/2406.07008v2
• Date: Sun, 19 Oct 2025 08:30:48 GMT
• Title: Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models
• Authors: Sooyeon Go, Kyungmook Choi, Minjung Shin, Youngjung Uh,
• Abstract summary: This paper tackles training-free appearance transfer, which produces an image with the structure of a target image from the appearance of a reference image.<n>Existing methods usually do not reflect semantic correspondence, as they rely on query-key similarity within the self-attention layer to establish correspondences between images.
• Score: 14.201696307616464
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As pre-trained text-to-image diffusion models have become a useful tool for image synthesis, people want to specify the results in various ways. This paper tackles training-free appearance transfer, which produces an image with the structure of a target image from the appearance of a reference image. Existing methods usually do not reflect semantic correspondence, as they rely on query-key similarity within the self-attention layer to establish correspondences between images. To this end, we propose explicitly rearranging the features according to the dense semantic correspondences. Extensive experiments show the superiority of our method in various aspects: preserving the structure of the target and reflecting the correct color from the reference, even when the two images are not aligned.

Related papers

• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• GenColor: Generative Color-Concept Association in Visual Design [1.9642328328398817]
  We introduce a generative approach for mining semantically resonant colors leveraging images generated by text-to-image models. The framework comprises three stages: concept instancing produces generative samples using diffusion models, text-guided image segmentation identifies concept-relevant regions within the image, and color association extracts primarily accompanied by accent colors.
  arXiv  Detail & Related papers  (2025-03-05T07:29:12Z)
• DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations [64.43387739794531]
  Current encoder-based approaches significantly impair the text controllability of text-to-image models while transferring styles. We introduce DEADiff to address this issue using the following two strategies. DEAiff attains the best visual stylization results and optimal balance between the text controllability inherent in the text-to-image model and style similarity to the reference image.
  arXiv  Detail & Related papers  (2024-03-11T17:35:23Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• SPColor: Semantic Prior Guided Exemplar-based Image Colorization [14.191819767895867]
  We propose SPColor, a semantic prior guided-based image colorization framework. SPColor first coarsely classifies pixels of the reference and target images to several pseudo-classes under the guidance of semantic prior. Our model outperforms recent state-of-the-art methods both quantitatively and qualitatively on public dataset.
  arXiv  Detail & Related papers  (2023-04-13T04:21:45Z)
• Describing Sets of Images with Textual-PCA [89.46499914148993]
  We seek to semantically describe a set of images, capturing both the attributes of single images and the variations within the set. Our procedure is analogous to Principle Component Analysis, in which the role of projection vectors is replaced with generated phrases.
  arXiv  Detail & Related papers  (2022-10-21T17:10:49Z)
• Marginal Contrastive Correspondence for Guided Image Generation [58.0605433671196]
  Exemplar-based image translation establishes dense correspondences between a conditional input and an exemplar from two different domains. Existing work builds the cross-domain correspondences implicitly by minimizing feature-wise distances across the two domains. We design a Marginal Contrastive Learning Network (MCL-Net) that explores contrastive learning to learn domain-invariant features for realistic exemplar-based image translation.
  arXiv  Detail & Related papers  (2022-04-01T13:55:44Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Learning to Compose Hypercolumns for Visual Correspondence [57.93635236871264]
  We introduce a novel approach to visual correspondence that dynamically composes effective features by leveraging relevant layers conditioned on the images to match. The proposed method, dubbed Dynamic Hyperpixel Flow, learns to compose hypercolumn features on the fly by selecting a small number of relevant layers from a deep convolutional neural network.
  arXiv  Detail & Related papers  (2020-07-21T04:03:22Z)
• Cross-domain Correspondence Learning for Exemplar-based Image Translation [59.35767271091425]
  We present a framework for exemplar-based image translation, which synthesizes a photo-realistic image from the input in a distinct domain. The output has the style (e.g., color, texture) in consistency with the semantically corresponding objects in the exemplar. We show that our method is superior to state-of-the-art methods in terms of image quality significantly.
  arXiv  Detail & Related papers  (2020-04-12T09:10:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.