Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models

• URL: http://arxiv.org/abs/2502.07466v1
• Date: Tue, 11 Feb 2025 11:17:39 GMT
• Title: Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models
• Authors: Lin Zhu, Xinbing Wang, Chenghu Zhou, Qinying Gu, Nanyang Ye,
• Abstract summary: We propose a masking-based method that efficiently decouples content and style from style-reference images. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle.
• Score: 44.4106999443933
• License:
• Abstract: Given a style-reference image as the additional image condition, text-to-image diffusion models have demonstrated impressive capabilities in generating images that possess the content of text prompts while adopting the visual style of the reference image. However, current state-of-the-art methods often struggle to disentangle content and style from style-reference images, leading to issues such as content leakages. To address this issue, we propose a masking-based method that efficiently decouples content from style without the need of tuning any model parameters. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle: guiding with appropriately-selected fewer conditions (e.g., dropping several image feature elements) can efficiently avoid unwanted content flowing into the diffusion models, enhancing the style transfer performances of text-to-image diffusion models. In this paper, we validate this finding both theoretically and experimentally. Extensive experiments across various styles demonstrate the effectiveness of our masking-based method and support our theoretical results.

Related papers

• Content-style disentangled representation for controllable artistic image stylization and generation [0.0]
  Controllable artistic image stylization and generation aims to render the content provided by text or image with the learned artistic style. This paper proposes a content-style representation disentangling method for controllable artistic image stylization and generation.
  arXiv  Detail & Related papers  (2024-12-19T03:42:58Z)
• Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics [3.9717825324709413]
  Style has been primarily considered in terms of artistic elements such as colors, brushstrokes, and lighting. In this study, we propose a zero-shot scheme for image variation with coordinated semantics.
  arXiv  Detail & Related papers  (2024-10-24T08:34:57Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• Improving Diffusion Models for Authentic Virtual Try-on in the Wild [53.96244595495942]
  This paper considers image-based virtual try-on, which renders an image of a person wearing a curated garment. We propose a novel diffusion model that improves garment fidelity and generates authentic virtual try-on images. We present a customization method using a pair of person-garment images, which significantly improves fidelity and authenticity.
  arXiv  Detail & Related papers  (2024-03-08T08:12:18Z)
• ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors [105.37795139586075]
  We propose a new task for stylizing'' text-to-image models, namely text-driven stylized image generation. We present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network. Experiments demonstrate the effectiveness of our ControlStyle in producing more visually pleasing and artistic results.
  arXiv  Detail & Related papers  (2023-11-09T15:50:52Z)
• Zero-Shot Contrastive Loss for Text-Guided Diffusion Image Style Transfer [38.957512116073616]
  We propose a zero-shot contrastive loss for diffusion models that doesn't require additional fine-tuning or auxiliary networks. Our method can generate images with the same semantic content as the source image in a zero-shot manner.
  arXiv  Detail & Related papers  (2023-03-15T13:47:02Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• Diffusion-based Image Translation using Disentangled Style and Content Representation [51.188396199083336]
  Diffusion-based image translation guided by semantic texts or a single target image has enabled flexible style transfer. It is often difficult to maintain the original content of the image during the reverse diffusion. We present a novel diffusion-based unsupervised image translation method using disentangled style and content representation. Our experimental results show that the proposed method outperforms state-of-the-art baseline models in both text-guided and image-guided translation tasks.
  arXiv  Detail & Related papers  (2022-09-30T06:44:37Z)
• Saliency Constrained Arbitrary Image Style Transfer using SIFT and DCNN [22.57205921266602]
  When common neural style transfer methods are used, the textures and colors in the style image are usually transferred imperfectly to the content image. This paper proposes a novel saliency constrained method to reduce or avoid such effects. The experiments show that the saliency maps of source images can help find the correct matching and avoid artifacts.
  arXiv  Detail & Related papers  (2022-01-14T09:00:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.