Real-time Identity Defenses against Malicious Personalization of Diffusion Models

• URL: http://arxiv.org/abs/2412.09844v2
• Date: Sun, 19 Jan 2025 15:03:39 GMT
• Title: Real-time Identity Defenses against Malicious Personalization of Diffusion Models
• Authors: Hanzhong Guo, Shen Nie, Chao Du, Tianyu Pang, Hao Sun, Chongxuan Li,
• Abstract summary: This study introduces the Real-time Identity Defender (RID), a neural network designed to generate adversarial perturbations through a single forward pass. RID achieves unprecedented efficiency, with defense times as low as 0.12 seconds on a single NVIDIA A100 80G GPU. Our model is envisioned to play a crucial role in safeguarding portrait rights, thereby preventing illegal and unethical uses.
• Score: 39.861209610456356
• License:
• Abstract: Personalized generative diffusion models, capable of synthesizing highly realistic images based on a few reference portraits, may pose substantial social, ethical, and legal risks via identity replication. Existing defense mechanisms rely on computationally intensive adversarial perturbations tailored to individual images, rendering them impractical for real-world deployment. This study introduces the Real-time Identity Defender (RID), a neural network designed to generate adversarial perturbations through a single forward pass, bypassing the need for image-specific optimization. RID achieves unprecedented efficiency, with defense times as low as 0.12 seconds on a single NVIDIA A100 80G GPU (4,400 times faster than leading methods) and 1.1 seconds per image on a standard Intel i9 CPU, making it suitable for edge devices such as smartphones. Despite its efficiency, RID achieves promising protection performance across visual and quantitative benchmarks, effectively mitigating identity replication risks. Our analysis reveals that RID's perturbations mimic the efficacy of traditional defenses while exhibiting properties distinct from natural noise, such as Gaussian perturbations. To enhance robustness, we extend RID into an ensemble framework that integrates multiple pre-trained text-to-image diffusion models, ensuring resilience against black-box attacks and post-processing techniques, including image compression and purification. Our model is envisioned to play a crucial role in safeguarding portrait rights, thereby preventing illegal and unethical uses.

Related papers

• Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models [9.548195579003897]
  We introduce pre-training to reduce latency and propose a mixture-of-perturbations approach to minimize performance degradation. Our novel training strategy computes protection loss across multiple VAE feature spaces, while adaptive targeted protection at inference enhances robustness. Experiments show comparable protection performance with improved invisibility and drastically reduced inference time.
  arXiv  Detail & Related papers  (2024-12-16T03:46:45Z)
• Pixel Is Not a Barrier: An Effective Evasion Attack for Pixel-Domain Diffusion Models [9.905296922309157]
  Diffusion Models have emerged as powerful generative models for high-quality image synthesis, with many subsequent image editing techniques based on them. Previous works have attempted to safeguard images from diffusion-based editing by adding imperceptible perturbations. Our work proposes a novel attack framework, AtkPDM, which exploits vulnerabilities in denoising UNets and a latent optimization strategy to enhance the naturalness of adversarial images.
  arXiv  Detail & Related papers  (2024-08-21T17:56:34Z)
• Latent Feature and Attention Dual Erasure Attack against Multi-View Diffusion Models for 3D Assets Protection [5.677981100052122]
  Multi-View Diffusion Models (MVDMs) enable remarkable improvements in the field of 3D geometric reconstruction. This paper is the first to address the intellectual property infringement issue arising from MVDMs.
  arXiv  Detail & Related papers  (2024-08-21T08:06:06Z)
• StealthDiffusion: Towards Evading Diffusion Forensic Detection through Diffusion Model [62.25424831998405]
  StealthDiffusion is a framework that modifies AI-generated images into high-quality, imperceptible adversarial examples. It is effective in both white-box and black-box settings, transforming AI-generated images into high-quality adversarial forgeries.
  arXiv  Detail & Related papers  (2024-08-11T01:22:29Z)
• DDAP: Dual-Domain Anti-Personalization against Text-to-Image Diffusion Models [18.938687631109925]
  Diffusion-based personalized visual content generation technologies have achieved significant breakthroughs. However, when misused to fabricate fake news or unsettling content targeting individuals, these technologies could cause considerable societal harm. This paper introduces a novel Dual-Domain Anti-Personalization framework (DDAP) By alternating between these two methods, we construct the DDAP framework, effectively harnessing the strengths of both domains.
  arXiv  Detail & Related papers  (2024-07-29T16:11:21Z)
• Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model [61.53213964333474]
  We propose a unified framework Adv-Diffusion that can generate imperceptible adversarial identity perturbations in the latent space but not the raw pixel space. Specifically, we propose the identity-sensitive conditioned diffusion generative model to generate semantic perturbations in the surroundings. The designed adaptive strength-based adversarial perturbation algorithm can ensure both attack transferability and stealthiness.
  arXiv  Detail & Related papers  (2023-12-18T15:25:23Z)
• PortraitBooth: A Versatile Portrait Model for Fast Identity-preserved Personalization [92.90392834835751]
  PortraitBooth is designed for high efficiency, robust identity preservation, and expression-editable text-to-image generation. PortraitBooth eliminates computational overhead and mitigates identity distortion. It incorporates emotion-aware cross-attention control for diverse facial expressions in generated images.
  arXiv  Detail & Related papers  (2023-12-11T13:03:29Z)
• DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection [64.77548539959501]
  DiffProtect produces more natural-looking encrypted images than state-of-the-art methods. It achieves significantly higher attack success rates, e.g., 24.5% and 25.1% absolute improvements on the CelebA-HQ and FFHQ datasets.
  arXiv  Detail & Related papers  (2023-05-23T02:45:49Z)
• Data Forensics in Diffusion Models: A Systematic Analysis of Membership Privacy [62.16582309504159]
  We develop a systematic analysis of membership inference attacks on diffusion models and propose novel attack methods tailored to each attack scenario. Our approach exploits easily obtainable quantities and is highly effective, achieving near-perfect attack performance (>0.9 AUCROC) in realistic scenarios.
  arXiv  Detail & Related papers  (2023-02-15T17:37:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.