DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection

• URL: http://arxiv.org/abs/2405.09882v1
• Date: Thu, 16 May 2024 08:05:36 GMT
• Title: DiffAM: Diffusion-based Adversarial Makeup Transfer for Facial Privacy Protection
• Authors: Yuhao Sun, Lingyun Yu, Hongtao Xie, Jiaming Li, Yongdong Zhang,
• Abstract summary: DiffAM is a novel approach to generate high-quality protected face images with adversarial makeup transferred from reference images. Experiments demonstrate that DiffAM achieves higher visual quality and attack success rates with a gain of 12.98% under black-box setting.
• Score: 60.73609509756533
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid development of face recognition (FR) systems, the privacy of face images on social media is facing severe challenges due to the abuse of unauthorized FR systems. Some studies utilize adversarial attack techniques to defend against malicious FR systems by generating adversarial examples. However, the generated adversarial examples, i.e., the protected face images, tend to suffer from subpar visual quality and low transferability. In this paper, we propose a novel face protection approach, dubbed DiffAM, which leverages the powerful generative ability of diffusion models to generate high-quality protected face images with adversarial makeup transferred from reference images. To be specific, we first introduce a makeup removal module to generate non-makeup images utilizing a fine-tuned diffusion model with guidance of textual prompts in CLIP space. As the inverse process of makeup transfer, makeup removal can make it easier to establish the deterministic relationship between makeup domain and non-makeup domain regardless of elaborate text prompts. Then, with this relationship, a CLIP-based makeup loss along with an ensemble attack strategy is introduced to jointly guide the direction of adversarial makeup domain, achieving the generation of protected face images with natural-looking makeup and high black-box transferability. Extensive experiments demonstrate that DiffAM achieves higher visual quality and attack success rates with a gain of 12.98% under black-box setting compared with the state of the arts. The code will be available at https://github.com/HansSunY/DiffAM.

Related papers

• DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing [93.45507533317405]
  DiffusionGuard is a robust and effective defense method against unauthorized edits by diffusion-based image editing models. We introduce a novel objective that generates adversarial noise targeting the early stage of the diffusion process. We also introduce a mask-augmentation technique to enhance robustness against various masks during test time.
  arXiv  Detail & Related papers  (2024-10-08T05:19:19Z)
• Makeup-Guided Facial Privacy Protection via Untrained Neural Network Priors [20.006182211662853]
  Deep learning-based face recognition systems pose significant privacy risks by tracking users without their consent. Recent facial privacy protection approaches advocate embedding adversarial noise into the natural looking makeup styles. These approaches require training on large-scale makeup datasets that are not always readily available. We propose a test-time optimization approach that solely optimize an untrained neural network to transfer makeup style from a reference to a source image in an adversarial manner.
  arXiv  Detail & Related papers  (2024-08-20T17:59:39Z)
• Imperceptible Face Forgery Attack via Adversarial Semantic Mask [59.23247545399068]
  We propose an Adversarial Semantic Mask Attack framework (ASMA) which can generate adversarial examples with good transferability and invisibility. Specifically, we propose a novel adversarial semantic mask generative model, which can constrain generated perturbations in local semantic regions for good stealthiness.
  arXiv  Detail & Related papers  (2024-06-16T10:38:11Z)
• 3D-Aware Adversarial Makeup Generation for Facial Privacy Protection [23.915259014651337]
  3D-Aware Adversarial Makeup Generation GAN (3DAM-GAN) A UV-based generator consisting of a novel Makeup Adjustment Module (MAM) and Makeup Transfer Module (MTM) is designed to render realistic and robust makeup. Experiment results on several benchmark datasets demonstrate that 3DAM-GAN could effectively protect faces against various FR models.
  arXiv  Detail & Related papers  (2023-06-26T12:27:59Z)
• DiffProtect: Generate Adversarial Examples with Diffusion Models for Facial Privacy Protection [64.77548539959501]
  DiffProtect produces more natural-looking encrypted images than state-of-the-art methods. It achieves significantly higher attack success rates, e.g., 24.5% and 25.1% absolute improvements on the CelebA-HQ and FFHQ datasets.
  arXiv  Detail & Related papers  (2023-05-23T02:45:49Z)
• Attribute-Guided Encryption with Facial Texture Masking [64.77548539959501]
  We propose Attribute Guided Encryption with Facial Texture Masking to protect users from unauthorized facial recognition systems. Our proposed method produces more natural-looking encrypted images than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-22T23:50:43Z)
• Restricted Black-box Adversarial Attack Against DeepFake Face Swapping [70.82017781235535]
  We introduce a practical adversarial attack that does not require any queries to the facial image forgery model. Our method is built on a substitute model persuing for face reconstruction and then transfers adversarial examples from the substitute model directly to inaccessible black-box DeepFake models.
  arXiv  Detail & Related papers  (2022-04-26T14:36:06Z)
• Protecting Facial Privacy: Generating Adversarial Identity Masks via Style-robust Makeup Transfer [24.25863892897547]
  adversarial makeup transfer GAN (AMT-GAN) is a novel face protection method aiming at constructing adversarial face images. In this paper, we introduce a new regularization module along with a joint training strategy to reconcile the conflicts between the adversarial noises and the cycle consistence loss in makeup transfer.
  arXiv  Detail & Related papers  (2022-03-07T03:56:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.