SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective

• URL: http://arxiv.org/abs/2303.09270v3
• Date: Thu, 2 Nov 2023 18:03:19 GMT
• Title: SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective
• Abstract summary: We propose SpectralCLIP, which is based on a spectral representation of the CLIP embedding sequence. We show that SpectralCLIP prevents the generation of artifacts effectively in quantitative and qualitative terms. We also apply SpectralCLIP to text-conditioned image generation and show that it prevents written words in the generated images.
• Score: 60.60970143677462
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Owing to the power of vision-language foundation models, e.g., CLIP, the area of image synthesis has seen recent important advances. Particularly, for style transfer, CLIP enables transferring more general and abstract styles without collecting the style images in advance, as the style can be efficiently described with natural language, and the result is optimized by minimizing the CLIP similarity between the text description and the stylized image. However, directly using CLIP to guide style transfer leads to undesirable artifacts (mainly written words and unrelated visual entities) spread over the image. In this paper, we propose SpectralCLIP, which is based on a spectral representation of the CLIP embedding sequence, where most of the common artifacts occupy specific frequencies. By masking the band including these frequencies, we can condition the generation process to adhere to the target style properties (e.g., color, texture, paint stroke, etc.) while excluding the generation of larger-scale structures corresponding to the artifacts. Experimental results show that SpectralCLIP prevents the generation of artifacts effectively in quantitative and qualitative terms, without impairing the stylisation quality. We also apply SpectralCLIP to text-conditioned image generation and show that it prevents written words in the generated images. Our code is available at https://github.com/zipengxuc/SpectralCLIP.

Related papers

• Fine-grained CLIP fine-tuning with self-annotated region alignment [57.28808560528696]
  Contrastive Language-Image Pre-training has been shown to have limitations in its fine-grained dense feature representation.<n>We propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP) to boost the fine-grained representation ability in the CLIP fine-tuning.
  arXiv  Detail & Related papers  (2026-07-15T10:06:39Z)
• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• Implicit Inversion turns CLIP into a Decoder [15.428694454730541]
  We show that image synthesis is possible using CLIP alone -- without any decoder, training, or fine-tuning.<n>Our approach optimize a frequency-aware implicit neural representation that encourages coarse-to-fine generation by stratifying across network layers.<n>Without altering CLIP's weights, this framework unlocks capabilities such as text-to-image generation, style transfer, and image reconstruction.
  arXiv  Detail & Related papers  (2025-05-29T06:55:26Z)
• Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP [44.90909692140324]
  We propose a Gradient-based visual and textual Explanation method for CLIP (Grad-ECLIP) We produce high-quality visual explanations by applying channel and spatial weights on token features. We also propose an application with Grad-ECLIP to boost the fine-grained alignment in the CLIP fine-tuning.
  arXiv  Detail & Related papers  (2025-02-26T04:50:20Z)
• Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation [26.57958479362817]
  Control-CLIP is a novel decoupled CLIP fine-tuning framework.<n>It enables the CLIP model to learn the meaning of category and style in a complement manner.<n>Experiments across multiple domains confirm the effectiveness of our approach.
  arXiv  Detail & Related papers  (2025-02-17T08:03:55Z)
• Dissecting CLIP: Decomposition with a Schur Complement-based Approach [8.056359341994941]
  We extend the application of CLIP embeddings to quantify and interpret the intrinsic diversity of text-to-image models. We demonstrate the use of the Schur complement-based decomposition to nullify the influence of a given prompt in the CLIP embedding of an image.
  arXiv  Detail & Related papers  (2024-12-24T18:07:57Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Finetuning CLIP to Reason about Pairwise Differences [52.028073305958074]
  We propose an approach to train vision-language models such as CLIP in a contrastive manner to reason about differences in embedding space. We first demonstrate that our approach yields significantly improved capabilities in ranking images by a certain attribute. We also illustrate that the resulting embeddings obey a larger degree of geometric properties in embedding space.
  arXiv  Detail & Related papers  (2024-09-15T13:02:14Z)
• Interpreting CLIP's Image Representation via Text-Based Decomposition [73.54377859089801]
  We investigate the CLIP image encoder by analyzing how individual model components affect the final representation. We decompose the image representation as a sum across individual image patches, model layers, and attention heads. We use this understanding to remove spurious features from CLIP and to create a strong zero-shot image segmenter.
  arXiv  Detail & Related papers  (2023-10-09T17:59:04Z)
• CLIP2GAN: Towards Bridging Text with the Latent Space of GANs [128.47600914674985]
  We propose a novel framework, i.e., CLIP2GAN, by leveraging CLIP model and StyleGAN. The key idea of our CLIP2GAN is to bridge the output feature embedding space of CLIP and the input latent space of StyleGAN.
  arXiv  Detail & Related papers  (2022-11-28T04:07:17Z)
• CLIP-PAE: Projection-Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation [4.078926358349661]
  Contrastive Language-Image Pre-Training (CLIP) bridges images and text by embedding them into a joint latent space. Due to the discrepancy between image and text embeddings in the joint space, using text embeddings as the optimization target often introduces undesired artifacts in the resulting images. We introduce CLIP Projection-Augmentation Embedding (PAE) as an optimization target to improve the performance of text-guided image manipulation.
  arXiv  Detail & Related papers  (2022-10-08T05:12:25Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.