Content-style disentangled representation for controllable artistic image stylization and generation

• URL: http://arxiv.org/abs/2412.14496v1
• Date: Thu, 19 Dec 2024 03:42:58 GMT
• Title: Content-style disentangled representation for controllable artistic image stylization and generation
• Authors: Ma Zhuoqi, Zhang Yixuan, You Zejun, Tian Long, Liu Xiyang,
• Abstract summary: Controllable artistic image stylization and generation aims to render the content provided by text or image with the learned artistic style. This paper proposes a content-style representation disentangling method for controllable artistic image stylization and generation.
• Score: 0.0
• License:
• Abstract: Controllable artistic image stylization and generation aims to render the content provided by text or image with the learned artistic style, where content and style decoupling is the key to achieve satisfactory results. However, current methods for content and style disentanglement primarily rely on image information for supervision, which leads to two problems: 1) models can only support one modality for style or content input;2) incomplete disentanglement resulting in semantic interference from the reference image. To address the above issues, this paper proposes a content-style representation disentangling method for controllable artistic image stylization and generation. We construct a WikiStyle+ dataset consists of artworks with corresponding textual descriptions for style and content. Based on the multimodal dataset, we propose a disentangled content and style representations guided diffusion model. The disentangled representations are first learned by Q-Formers and then injected into a pre-trained diffusion model using learnable multi-step cross-attention layers for better controllable stylization. This approach allows model to accommodate inputs from different modalities. Experimental results show that our method achieves a thorough disentanglement of content and style in reference images under multimodal supervision, thereby enabling a harmonious integration of content and style in the generated outputs, successfully producing style-consistent and expressive stylized images.

Related papers

• Less is More: Masking Elements in Image Condition Features Avoids Content Leakages in Style Transfer Diffusion Models [44.4106999443933]
  We propose a masking-based method that efficiently decouples content and style from style-reference images. By simply masking specific elements in the style reference's image features, we uncover a critical yet under-explored principle.
  arXiv  Detail & Related papers  (2025-02-11T11:17:39Z)
• DiffuseST: Unleashing the Capability of the Diffusion Model for Style Transfer [13.588643982359413]
  Style transfer aims to fuse the artistic representation of a style image with the structural information of a content image. Existing methods train specific networks or utilize pre-trained models to learn content and style features. We propose a novel and training-free approach for style transfer, combining textual embedding with spatial features.
  arXiv  Detail & Related papers  (2024-10-19T06:42:43Z)
• Customizing Text-to-Image Models with a Single Image Pair [47.49970731632113]
  Art reinterpretation is the practice of creating a variation of a reference work, making a paired artwork that exhibits a distinct artistic style. We propose Pair Customization, a new customization method that learns stylistic difference from a single image pair and then applies the acquired style to the generation process.
  arXiv  Detail & Related papers  (2024-05-02T17:59:52Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors [105.37795139586075]
  We propose a new task for stylizing'' text-to-image models, namely text-driven stylized image generation. We present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network. Experiments demonstrate the effectiveness of our ControlStyle in producing more visually pleasing and artistic results.
  arXiv  Detail & Related papers  (2023-11-09T15:50:52Z)
• StyleAdapter: A Unified Stylized Image Generation Model [97.24936247688824]
  StyleAdapter is a unified stylized image generation model capable of producing a variety of stylized images. It can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet.
  arXiv  Detail & Related papers  (2023-09-04T19:16:46Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.