DiffArtist: Towards Structure and Appearance Controllable Image Stylization

• URL: http://arxiv.org/abs/2407.15842v3
• Date: Wed, 23 Apr 2025 17:46:08 GMT
• Title: DiffArtist: Towards Structure and Appearance Controllable Image Stylization
• Authors: Ruixiang Jiang, Changwen Chen,
• Abstract summary: We present a comprehensive study on the simultaneous stylization of structure and appearance of 2D images.<n>We introduce DiffArtist, which is the first stylization method to allow for dual controllability over structure and appearance.
• Score: 19.5597806965592
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artistic style includes both structural and appearance elements. Existing neural stylization techniques primarily focus on transferring appearance features such as color and texture, often neglecting the equally crucial aspect of structural stylization. In this paper, we present a comprehensive study on the simultaneous stylization of structure and appearance of 2D images. Specifically, we introduce DiffArtist, which, to the best of our knowledge, is the first stylization method to allow for dual controllability over structure and appearance. Our key insight is to represent structure and appearance as separate diffusion processes to achieve complete disentanglement without requiring any training, thereby endowing users with unprecedented controllability for both components. The evaluation of stylization of both appearance and structure, however, remains challenging as it necessitates semantic understanding. To this end, we further propose a Multimodal LLM-based style evaluator, which better aligns with human preferences than metrics lacking semantic understanding. With this powerful evaluator, we conduct extensive analysis, demonstrating that DiffArtist achieves superior style fidelity, editability, and structure-appearance disentanglement. These merits make DiffArtist a highly versatile solution for creative applications. Project homepage: https://github.com/songrise/Artist.

Related papers

• Compose Your Aesthetics: Empowering Text-to-Image Models with the Principles of Art [61.28133495240179]
  We propose a novel task of aesthetics alignment which seeks to align user-specified aesthetics with the T2I generation output. Inspired by how artworks provide an invaluable perspective to approach aesthetics, we codify visual aesthetics using the compositional framework artists employ. We demonstrate that T2I DMs can effectively offer 10 compositional controls through user-specified PoA conditions.
  arXiv  Detail & Related papers  (2025-03-15T06:58:09Z)
• StyleBlend: Enhancing Style-Specific Content Creation in Text-to-Image Diffusion Models [10.685779311280266]
  StyleBlend is a method designed to learn and apply style representations from a limited set of reference images. Our approach decomposes style into two components, composition and texture, each learned through different strategies.
  arXiv  Detail & Related papers  (2025-02-13T08:26:54Z)
• IntroStyle: Training-Free Introspective Style Attribution using Diffusion Features [89.95303251220734]
  We present a training-free framework to solve the style attribution problem, using the features produced by a diffusion model alone. This is denoted as introspective style attribution (IntroStyle) and demonstrates superior performance to state-of-the-art models for style retrieval. We also introduce a synthetic dataset of Style Hacks (SHacks) to isolate artistic style and evaluate fine-grained style attribution performance.
  arXiv  Detail & Related papers  (2024-12-19T01:21:23Z)
• Learning Artistic Signatures: Symmetry Discovery and Style Transfer [8.288443063900825]
  There is no undisputed definition of artistic style. Style should be thought of as a set of global symmetries that dictate the arrangement of local textures. We show that by considering both local and global features, using both Lie generators and traditional measures of texture, we can quantitatively capture the stylistic similarity between artists better than with either set of features alone.
  arXiv  Detail & Related papers  (2024-12-05T18:56:23Z)
• DiffuseST: Unleashing the Capability of the Diffusion Model for Style Transfer [13.588643982359413]
  Style transfer aims to fuse the artistic representation of a style image with the structural information of a content image. Existing methods train specific networks or utilize pre-trained models to learn content and style features. We propose a novel and training-free approach for style transfer, combining textual embedding with spatial features.
  arXiv  Detail & Related papers  (2024-10-19T06:42:43Z)
• StyleBrush: Style Extraction and Transfer from a Single Image [19.652575295703485]
  Stylization for visual content aims to add specific style patterns at the pixel level while preserving the original structural features. We propose StyleBrush, a method that accurately captures styles from a reference image and brushes'' the extracted style onto other input visual content.
  arXiv  Detail & Related papers  (2024-08-18T14:27:20Z)
• InstantStyle-Plus: Style Transfer with Content-Preserving in Text-to-Image Generation [4.1177497612346]
  Style transfer is an inventive process designed to create an image that maintains the essence of the original while embracing the visual style of another. We introduce InstantStyle-Plus, an approach that prioritizes the integrity of the original content while seamlessly integrating the target style.
  arXiv  Detail & Related papers  (2024-06-30T18:05:33Z)
• ArtWeaver: Advanced Dynamic Style Integration via Diffusion Model [73.95608242322949]
  Stylized Text-to-Image Generation (STIG) aims to generate images from text prompts and style reference images. We present ArtWeaver, a novel framework that leverages pretrained Stable Diffusion to address challenges such as misinterpreted styles and inconsistent semantics.
  arXiv  Detail & Related papers  (2024-05-24T07:19:40Z)
• Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt [12.27693060663517]
  Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. We propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST. Our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.
  arXiv  Detail & Related papers  (2024-04-17T15:28:53Z)
• StyleForge: Enhancing Text-to-Image Synthesis for Any Artistic Styles with Dual Binding [7.291687946822539]
  We introduce Single-StyleForge, a novel approach for personalized text-to-image synthesis across diverse artistic styles. We also present Multi-StyleForge, which enhances image quality and text alignment by binding multiple tokens to partial style attributes.
  arXiv  Detail & Related papers  (2024-04-08T07:43:23Z)
• Implicit Style-Content Separation using B-LoRA [61.664293840163865]
  We introduce B-LoRA, a method that implicitly separate the style and content components of a single image. By analyzing the architecture of SDXL combined with LoRA, we find that jointly learning the LoRA weights of two specific blocks achieves style-content separation.
  arXiv  Detail & Related papers  (2024-03-21T17:20:21Z)
• Style Aligned Image Generation via Shared Attention [61.121465570763085]
  We introduce StyleAligned, a technique designed to establish style alignment among a series of generated images. By employing minimal attention sharing' during the diffusion process, our method maintains style consistency across images within T2I models. Our method's evaluation across diverse styles and text prompts demonstrates high-quality and fidelity.
  arXiv  Detail & Related papers  (2023-12-04T18:55:35Z)
• StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style Adapter [78.75422651890776]
  StyleCrafter is a generic method that enhances pre-trained T2V models with a style control adapter. To promote content-style disentanglement, we remove style descriptions from the text prompt and extract style information solely from the reference image. StyleCrafter efficiently generates high-quality stylized videos that align with the content of the texts and resemble the style of the reference images.
  arXiv  Detail & Related papers  (2023-12-01T03:53:21Z)
• $Z^*$: Zero-shot Style Transfer via Attention Rearrangement [27.185432348397693]
  This study shows that vanilla diffusion models can directly extract style information and seamlessly integrate the generative prior into the content image without retraining. We adopt dual denoising paths to represent content/style references in latent space and then guide the content image denoising process with style latent codes.
  arXiv  Detail & Related papers  (2023-11-25T11:03:43Z)
• ControlStyle: Text-Driven Stylized Image Generation Using Diffusion Priors [105.37795139586075]
  We propose a new task for stylizing'' text-to-image models, namely text-driven stylized image generation. We present a new diffusion model (ControlStyle) via upgrading a pre-trained text-to-image model with a trainable modulation network. Experiments demonstrate the effectiveness of our ControlStyle in producing more visually pleasing and artistic results.
  arXiv  Detail & Related papers  (2023-11-09T15:50:52Z)
• ALADIN-NST: Self-supervised disentangled representation learning of artistic style through Neural Style Transfer [60.6863849241972]
  We learn a representation of visual artistic style more strongly disentangled from the semantic content depicted in an image. We show that strongly addressing the disentanglement of style and content leads to large gains in style-specific metrics.
  arXiv  Detail & Related papers  (2023-04-12T10:33:18Z)
• Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis [78.28620571530706]
  Large-scale diffusion models have achieved state-of-the-art results on text-to-image synthesis (T2I) tasks. We improve the compositional skills of T2I models, specifically more accurate attribute binding and better image compositions.
  arXiv  Detail & Related papers  (2022-12-09T18:30:24Z)
• DiffStyler: Controllable Dual Diffusion for Text-Driven Image Stylization [66.42741426640633]
  DiffStyler is a dual diffusion processing architecture to control the balance between the content and style of diffused results. We propose a content image-based learnable noise on which the reverse denoising process is based, enabling the stylization results to better preserve the structure information of the content image.
  arXiv  Detail & Related papers  (2022-11-19T12:30:44Z)
• Arbitrary Style Transfer with Structure Enhancement by Combining the Global and Local Loss [51.309905690367835]
  We introduce a novel arbitrary style transfer method with structure enhancement by combining the global and local loss. Experimental results demonstrate that our method can generate higher-quality images with impressive visual effects.
  arXiv  Detail & Related papers  (2022-07-23T07:02:57Z)
• Domain Enhanced Arbitrary Image Style Transfer via Contrastive Learning [84.8813842101747]
  Contrastive Arbitrary Style Transfer (CAST) is a new style representation learning and style transfer method via contrastive learning. Our framework consists of three key components, i.e., a multi-layer style projector for style code encoding, a domain enhancement module for effective learning of style distribution, and a generative network for image style transfer.
  arXiv  Detail & Related papers  (2022-05-19T13:11:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.