ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank

• URL: http://arxiv.org/abs/2312.06135v1
• Date: Mon, 11 Dec 2023 05:53:40 GMT
• Title: ArtBank: Artistic Style Transfer with Pre-trained Diffusion Model and Implicit Style Prompt Bank
• Authors: Zhanjie Zhang, Quanwei Zhang, Guangyuan Li, Wei Xing, Lei Zhao, Jiakai Sun, Zehua Lan, Junsheng Luan, Yiling Huang, Huaizhong Lin
• Abstract summary: Artistic style transfer aims to repaint the content image with the learned artistic style. Existing artistic style transfer methods can be divided into two categories: small model-based approaches and pre-trained large-scale model-based approaches. We propose ArtBank, a novel artistic style transfer framework, to generate highly realistic stylized images.
• Score: 9.99530386586636
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Artistic style transfer aims to repaint the content image with the learned artistic style. Existing artistic style transfer methods can be divided into two categories: small model-based approaches and pre-trained large-scale model-based approaches. Small model-based approaches can preserve the content strucuture, but fail to produce highly realistic stylized images and introduce artifacts and disharmonious patterns; Pre-trained large-scale model-based approaches can generate highly realistic stylized images but struggle with preserving the content structure. To address the above issues, we propose ArtBank, a novel artistic style transfer framework, to generate highly realistic stylized images while preserving the content structure of the content images. Specifically, to sufficiently dig out the knowledge embedded in pre-trained large-scale models, an Implicit Style Prompt Bank (ISPB), a set of trainable parameter matrices, is designed to learn and store knowledge from the collection of artworks and behave as a visual prompt to guide pre-trained large-scale models to generate highly realistic stylized images while preserving content structure. Besides, to accelerate training the above ISPB, we propose a novel Spatial-Statistical-based self-Attention Module (SSAM). The qualitative and quantitative experiments demonstrate the superiority of our proposed method over state-of-the-art artistic style transfer methods.

Related papers

• Ada-adapter:Fast Few-shot Style Personlization of Diffusion Model with Pre-trained Image Encoder [57.574544285878794]
  Ada-Adapter is a novel framework for few-shot style personalization of diffusion models. Our method enables efficient zero-shot style transfer utilizing a single reference image. We demonstrate the effectiveness of our approach on various artistic styles, including flat art, 3D rendering, and logo design.
  arXiv  Detail & Related papers  (2024-07-08T02:00:17Z)
• MuseumMaker: Continual Style Customization without Catastrophic Forgetting [50.12727620780213]
  We propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner. When facing with a new customization style, we develop a style distillation loss module to extract and learn the styles of the training data for new image generation. It can minimize the learning biases caused by content of new training images, and address the catastrophic overfitting issue induced by few-shot images.
  arXiv  Detail & Related papers  (2024-04-25T13:51:38Z)
• Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt [12.27693060663517]
  Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. We propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST. Our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.
  arXiv  Detail & Related papers  (2024-04-17T15:28:53Z)
• Generative AI Model for Artistic Style Transfer Using Convolutional Neural Networks [0.0]
  Artistic style transfer involves fusing the content of one image with the artistic style of another to create unique visual compositions. This paper presents a comprehensive overview of a novel technique for style transfer using Convolutional Neural Networks (CNNs)
  arXiv  Detail & Related papers  (2023-10-27T16:21:17Z)
• DIFF-NST: Diffusion Interleaving For deFormable Neural Style Transfer [27.39248034592382]
  We propose using a new class of models to perform style transfer while enabling deformable style transfer. We show how leveraging the priors of these models can expose new artistic controls at inference time.
  arXiv  Detail & Related papers  (2023-07-09T12:13:43Z)
• Learning to Evaluate the Artness of AI-generated Images [64.48229009396186]
  ArtScore is a metric designed to evaluate the degree to which an image resembles authentic artworks by artists. We employ pre-trained models for photo and artwork generation, resulting in a series of mixed models. This dataset is then employed to train a neural network that learns to estimate quantized artness levels of arbitrary images.
  arXiv  Detail & Related papers  (2023-05-08T17:58:27Z)
• A Unified Arbitrary Style Transfer Framework via Adaptive Contrastive Learning [84.8813842101747]
  Unified Contrastive Arbitrary Style Transfer (UCAST) is a novel style representation learning and transfer framework. We present an adaptive contrastive learning scheme for style transfer by introducing an input-dependent temperature. Our framework consists of three key components, i.e., a parallel contrastive learning scheme for style representation and style transfer, a domain enhancement module for effective learning of style distribution, and a generative network for style transfer.
  arXiv  Detail & Related papers  (2023-03-09T04:35:00Z)
• Neural Artistic Style Transfer with Conditional Adversaria [0.0]
  A neural artistic style transformation model can modify the appearance of a simple image by adding the style of a famous image. In this paper, we present two methods that step toward the style image independent neural style transfer model. Our novel contribution is a unidirectional-GAN model that ensures the Cyclic consistency by the model architecture.
  arXiv  Detail & Related papers  (2023-02-08T04:34:20Z)
• QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity [94.5479418998225]
  We propose a new style transfer framework called QuantArt for high visual-fidelity stylization. Our framework achieves significantly higher visual fidelity compared with the existing style transfer methods.
  arXiv  Detail & Related papers  (2022-12-20T17:09:53Z)
• Inversion-Based Style Transfer with Diffusion Models [78.93863016223858]
  Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image.
  arXiv  Detail & Related papers  (2022-11-23T18:44:25Z)
• AesUST: Towards Aesthetic-Enhanced Universal Style Transfer [15.078430702469886]
  AesUST is a novel Aesthetic-enhanced Universal Style Transfer approach. We introduce an aesthetic discriminator to learn the universal human-delightful aesthetic features from a large corpus of artist-created paintings. We also develop a new two-stage transfer training strategy with two aesthetic regularizations to train our model more effectively.
  arXiv  Detail & Related papers  (2022-08-27T13:51:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.