Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer

• URL: http://arxiv.org/abs/2303.08622v2
• Date: Wed, 12 Apr 2023 14:17:00 GMT
• Title: Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer
• Authors: Serin Yang, Hyunmin Hwang, Jong Chul Ye
• Abstract summary: We propose a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks. Our method can generate images with the same semantic content as the source image in a zero-shot manner.
• Score: 38.957512116073616
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have shown great promise in text-guided image style transfer, but there is a trade-off between style transformation and content preservation due to their stochastic nature. Existing methods require computationally expensive fine-tuning of diffusion models or additional neural network. To address this, here we propose a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks. By leveraging patch-wise contrastive loss between generated samples and original image embeddings in the pre-trained diffusion model, our method can generate images with the same semantic content as the source image in a zero-shot manner. Our approach outperforms existing methods while preserving content and requiring no additional training, not only for image style transfer but also for image-to-image translation and manipulation. Our experimental results validate the effectiveness of our proposed method.

Related papers

• ReNoise: Real Image Inversion Through Iterative Noising [62.96073631599749]
  We introduce an inversion method with a high quality-to-operation ratio, enhancing reconstruction accuracy without increasing the number of operations. We evaluate the performance of our ReNoise technique using various sampling algorithms and models, including recent accelerated diffusion models.
  arXiv  Detail & Related papers  (2024-03-21T17:52:08Z)
• Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models [23.786473791344395]
  Cross-attention layers in diffusion models tend to disproportionately focus on certain tokens during the generation process. We introduce attention regulation, an on-the-fly optimization approach at inference time to align attention maps with the input text prompt. Experiment results show that our method consistently outperforms other baselines.
  arXiv  Detail & Related papers  (2024-03-11T02:18:27Z)
• Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer [19.355744690301403]
  We introduce a novel artistic style transfer method based on a pre-trained large-scale diffusion model without any optimization. Our experimental results demonstrate that our proposed method surpasses state-of-the-art methods in both conventional and diffusion-based style transfer baselines.
  arXiv  Detail & Related papers  (2023-12-11T09:53:12Z)
• Improving the Transferability of Adversarial Examples with Arbitrary Style Transfer [32.644062141738246]
  A style transfer network can alter the distribution of low-level visual features in an image while preserving semantic content for humans. We propose a novel attack method named Style Transfer Method (STM) that utilizes a proposed arbitrary style transfer network to transform the images into different domains. Our proposed method can significantly improve the adversarial transferability on either normally trained models or adversarially trained models.
  arXiv  Detail & Related papers  (2023-08-21T09:58:13Z)
• Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance [51.188396199083336]
  We present an approach that guides the reverse process of diffusion sampling by applying asymmetric gradient guidance. Our model's adaptability allows it to be implemented with both image-fusion and latent-dif models. Experiments show that our method outperforms various state-of-the-art models in image translation tasks.
  arXiv  Detail & Related papers  (2023-06-07T12:56:56Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• Diffusion-based Image Translation using Disentangled Style and Content Representation [51.188396199083336]
  Diffusion-based image translation guided by semantic texts or a single target image has enabled flexible style transfer. It is often difficult to maintain the original content of the image during the reverse diffusion. We present a novel diffusion-based unsupervised image translation method using disentangled style and content representation. Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.
  arXiv  Detail & Related papers  (2022-09-30T06:44:37Z)
• Encoding Robustness to Image Style via Adversarial Feature Perturbations [72.81911076841408]
  We adapt adversarial training by directly perturbing feature statistics, rather than image pixels, to produce robust models. Our proposed method, Adversarial Batch Normalization (AdvBN), is a single network layer that generates worst-case feature perturbations during training.
  arXiv  Detail & Related papers  (2020-09-18T17:52:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.