Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models

• URL: http://arxiv.org/abs/2408.10571v4
• Date: Thu, 10 Oct 2024 06:33:12 GMT
• Title: Prompt-Agnostic Adversarial Perturbation for Customized Diffusion Models
• Authors: Cong Wan, Yuhang He, Xiang Song, Yihong Gong,
• Abstract summary: We introduce a Prompt-Agnostic Adversarial Perturbation (PAP) method for customized diffusion models. PAP first models the prompt distribution using a Laplace Approximation, and then produces prompt-agnostic perturbations by maximizing a disturbance expectation. This approach effectively tackles the prompt-agnostic attacks, leading to improved defense stability.
• Score: 27.83772742404565
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have revolutionized customized text-to-image generation, allowing for efficient synthesis of photos from personal data with textual descriptions. However, these advancements bring forth risks including privacy breaches and unauthorized replication of artworks. Previous researches primarily center around using prompt-specific methods to generate adversarial examples to protect personal images, yet the effectiveness of existing methods is hindered by constrained adaptability to different prompts. In this paper, we introduce a Prompt-Agnostic Adversarial Perturbation (PAP) method for customized diffusion models. PAP first models the prompt distribution using a Laplace Approximation, and then produces prompt-agnostic perturbations by maximizing a disturbance expectation based on the modeled distribution. This approach effectively tackles the prompt-agnostic attacks, leading to improved defense stability. Extensive experiments in face privacy and artistic style protection, demonstrate the superior generalization of PAP in comparison to existing techniques. Our project page is available at https://github.com/vancyland/Prompt-Agnostic-Adversarial-Perturbation-for-Customized-Diffusion-Model s.github.io.

Related papers

• DiffPAD: Denoising Diffusion-based Adversarial Patch Decontamination [5.7254228484416325]
  DiffPAD is a novel framework that harnesses the power of diffusion models for adversarial patch decontamination. We show that DiffPAD achieves state-of-the-art adversarial robustness against patch attacks and also excels in recovering naturalistic images without patch remnants.
  arXiv  Detail & Related papers  (2024-10-31T15:09:36Z)
• DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models [18.938687631109925]
  Diffusion-based personalized visual content generation technologies have achieved significant breakthroughs. However, when misused to fabricate fake news or unsettling content targeting individuals, these technologies could cause considerable societal harm. This paper introduces a novel Dual-Domain Anti-Personalization framework (DDAP) By alternating between these two methods, we construct the DDAP framework, effectively harnessing the strengths of both domains.
  arXiv  Detail & Related papers  (2024-07-29T16:11:21Z)
• Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization [19.635385099376066]
  malicious users have misused diffusion-based customization methods like DreamBooth to create fake images. In this paper, we propose DisDiff, a novel adversarial attack method to disrupt the diffusion model outputs.
  arXiv  Detail & Related papers  (2024-05-31T02:45:31Z)
• Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesis [65.7968515029306]
  We propose a novel Coarse-to-Fine Latent Diffusion (CFLD) method for Pose-Guided Person Image Synthesis (PGPIS) A perception-refined decoder is designed to progressively refine a set of learnable queries and extract semantic understanding of person images as a coarse-grained prompt.
  arXiv  Detail & Related papers  (2024-02-28T06:07:07Z)
• Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model [61.53213964333474]
  We propose a unified framework Adv-Diffusion that can generate imperceptible adversarial identity perturbations in the latent space but not the raw pixel space. Specifically, we propose the identity-sensitive conditioned diffusion generative model to generate semantic perturbations in the surroundings. The designed adaptive strength-based adversarial perturbation algorithm can ensure both attack transferability and stealthiness.
  arXiv  Detail & Related papers  (2023-12-18T15:25:23Z)
• IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI [52.90082445349903]
  Diffusion-based image generation models can create artistic images that mimic the style of an artist or maliciously edit the original images for fake content. Several attempts have been made to protect the original images from such unauthorized data usage by adding imperceptible perturbations. In this work, we introduce a purification perturbation platform, named IMPRESS, to evaluate the effectiveness of imperceptible perturbations as a protective measure.
  arXiv  Detail & Related papers  (2023-10-30T03:33:41Z)
• Exposing the Fake: Effective Diffusion-Generated Images Detection [14.646957596560076]
  This paper proposes a novel detection method called Stepwise Error for Diffusion-generated Image Detection (SeDID) SeDID exploits the unique attributes of diffusion models, namely deterministic reverse and deterministic denoising errors. Our work makes a pivotal contribution to distinguishing diffusion model-generated images, marking a significant step in the domain of artificial intelligence security.
  arXiv  Detail & Related papers  (2023-07-12T16:16:37Z)
• Training Diffusion Models with Reinforcement Learning [82.29328477109826]
  Diffusion models are trained with an approximation to the log-likelihood objective. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for downstream objectives. We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms.
  arXiv  Detail & Related papers  (2023-05-22T17:57:41Z)
• Data Forensics in Diffusion Models: A Systematic Analysis of Membership Privacy [62.16582309504159]
  We develop a systematic analysis of membership inference attacks on diffusion models and propose novel attack methods tailored to each attack scenario. Our approach exploits easily obtainable quantities and is highly effective, achieving near-perfect attack performance (>0.9 AUCROC) in realistic scenarios.
  arXiv  Detail & Related papers  (2023-02-15T17:37:49Z)
• Federated Test-Time Adaptive Face Presentation Attack Detection with Dual-Phase Privacy Preservation [100.69458267888962]
  Face presentation attack detection (fPAD) plays a critical role in the modern face recognition pipeline. Due to legal and privacy issues, training data (real face images and spoof images) are not allowed to be directly shared between different data sources. We propose a Federated Test-Time Adaptive Face Presentation Attack Detection with Dual-Phase Privacy Preservation framework.
  arXiv  Detail & Related papers  (2021-10-25T02:51:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.