StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations

• URL: http://arxiv.org/abs/2505.18766v1
• Date: Sat, 24 May 2025 16:09:26 GMT
• Title: StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations
• Authors: Yanjie Li, Wenxuan Zhang, Xinqi Lyu, Yihao Liu, Bin Xiao,
• Abstract summary: Text-to-image diffusion models have been widely used for style mimicry and personalized customization.<n>Recent purification-based methods, such as DiffPure and Noise Upscaling, have successfully attacked these latest defenses.<n>We propose a novel anti-mimicry method, StyleGuard, to address these issues.
• Score: 27.678238166174115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, text-to-image diffusion models have been widely used for style mimicry and personalized customization through methods such as DreamBooth and Textual Inversion. This has raised concerns about intellectual property protection and the generation of deceptive content. Recent studies, such as Glaze and Anti-DreamBooth, have proposed using adversarial noise to protect images from these attacks. However, recent purification-based methods, such as DiffPure and Noise Upscaling, have successfully attacked these latest defenses, showing the vulnerabilities of these methods. Moreover, present methods show limited transferability across models, making them less effective against unknown text-to-image models. To address these issues, we propose a novel anti-mimicry method, StyleGuard. We propose a novel style loss that optimizes the style-related features in the latent space to make it deviate from the original image, which improves model-agnostic transferability. Additionally, to enhance the perturbation's ability to bypass diffusion-based purification, we designed a novel upscale loss that involves ensemble purifiers and upscalers during training. Extensive experiments on the WikiArt and CelebA datasets demonstrate that StyleGuard outperforms existing methods in robustness against various transformations and purifications, effectively countering style mimicry in various models. Moreover, StyleGuard is effective on different style mimicry methods, including DreamBooth and Textual Inversion.

Related papers

• Active Adversarial Noise Suppression for Image Forgery Localization [56.98050814363447]
  We introduce an Adversarial Noise Suppression Module (ANSM) that generate a defensive perturbation to suppress the attack effect of adversarial noise.<n>To our best knowledge, this is the first report of adversarial defense in image forgery localization tasks.
  arXiv  Detail & Related papers  (2025-06-15T14:53:27Z)
• SITA: Structurally Imperceptible and Transferable Adversarial Attacks for Stylized Image Generation [34.228338508482494]
  Current methods aimed at safeguarding artworks often employ adversarial attacks.<n>We propose a Structurally Imperceptible and Transferable Adrial (SITA) attacks.<n>It significantly outperforms existing methods in terms of transferability, computational efficiency, and noise imperceptibility.
  arXiv  Detail & Related papers  (2025-03-25T15:55:25Z)
• IntroStyle: Training-Free Introspective Style Attribution using Diffusion Features [89.95303251220734]
  We present a training-free framework to solve the style attribution problem.<n>IntroStyle is shown to have superior performance to state-of-the-art models for style attribution.
  arXiv  Detail & Related papers  (2024-12-19T01:21:23Z)
• Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation [24.381813317728195]
  Anti-Reference is a novel method that protects images from the threats posed by reference-based generation techniques.<n>We propose a unified loss function that enables joint attacks on fine-tuning-based customization methods.<n>Our method shows certain transfer attack capabilities, effectively challenging both gray-box models and some commercial APIs.
  arXiv  Detail & Related papers  (2024-12-08T16:04:45Z)
• DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing [93.45507533317405]
  DiffusionGuard is a robust and effective defense method against unauthorized edits by diffusion-based image editing models. We introduce a novel objective that generates adversarial noise targeting the early stage of the diffusion process. We also introduce a mask-augmentation technique to enhance robustness against various masks during test time.
  arXiv  Detail & Related papers  (2024-10-08T05:19:19Z)
• Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models [9.905296922309157]
  Diffusion Models have emerged as powerful generative models for high-quality image synthesis, with many subsequent image editing techniques based on them.<n>Previous works have attempted to safeguard images from diffusion-based editing by adding imperceptible perturbations.<n>Our work proposes a novel attack framework, AtkPDM, which exploits vulnerabilities in denoising UNets and a latent optimization strategy to enhance the naturalness of adversarial images.
  arXiv  Detail & Related papers  (2024-08-21T17:56:34Z)
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling [61.14140480095604]
  This paper presents an inversion-free portrait stylization framework based on diffusion models that accomplishes content and style feature fusion in merely four sampling steps. We propose a feature merging strategy to amalgamate redundant features in Consistency Features, thereby reducing the computational load of attention control.
  arXiv  Detail & Related papers  (2024-08-10T08:53:41Z)
• Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI [61.35083814817094]
  Several protection tools against style mimicry have been developed that incorporate small adversarial perturbations into artworks published online.<n>We find that low-effort and "off-the-shelf" techniques, such as image upscaling, are sufficient to create robust mimicry methods that significantly degrade existing protections.<n>We caution that tools based on adversarial perturbations cannot reliably protect artists from the misuse of generative AI.
  arXiv  Detail & Related papers  (2024-06-17T18:51:45Z)
• MuseumMaker: Continual Style Customization without Catastrophic Forgetting [50.12727620780213]
  We propose MuseumMaker, a method that enables the synthesis of images by following a set of customized styles in a never-end manner. When facing with a new customization style, we develop a style distillation loss module to extract and learn the styles of the training data for new image generation. It can minimize the learning biases caused by content of new training images, and address the catastrophic overfitting issue induced by few-shot images.
  arXiv  Detail & Related papers  (2024-04-25T13:51:38Z)
• DiffStyler: Diffusion-based Localized Image Style Transfer [0.0]
  Image style transfer aims to imbue digital imagery with the distinctive attributes of style targets, such as colors, brushstrokes, shapes. Despite the advancements in arbitrary style transfer methods, a prevalent challenge remains the delicate equilibrium between content semantics and style attributes. This paper introduces DiffStyler, a novel approach that facilitates efficient and precise arbitrary image style transfer.
  arXiv  Detail & Related papers  (2024-03-27T11:19:34Z)
• StyleAdv: Meta Style Adversarial Training for Cross-Domain Few-Shot Learning [89.86971464234533]
  Cross-Domain Few-Shot Learning (CD-FSL) is a recently emerging task that tackles few-shot learning across different domains. We propose a novel model-agnostic meta Style Adversarial training (StyleAdv) method together with a novel style adversarial attack method. Our method is gradually robust to the visual styles, thus boosting the generalization ability for novel target datasets.
  arXiv  Detail & Related papers  (2023-02-18T11:54:37Z)
• Towards Feature Space Adversarial Attack [18.874224858723494]
  We propose a new adversarial attack to Deep Neural Networks for image classification. Our attack focuses on perturbing abstract features, more specifically, features that denote styles. We show that our attack can generate adversarial samples that are more natural-looking than the state-of-the-art attacks.
  arXiv  Detail & Related papers  (2020-04-26T13:56:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.