FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution

• URL: http://arxiv.org/abs/2402.03705v1
• Date: Tue, 6 Feb 2024 04:56:43 GMT
• Title: FoolSDEdit: Deceptively Steering Your Edits Towards Targeted Attribute-aware Distribution
• Authors: Qi Zhou, Dongxia Wang, Tianlin Li, Zhihong Xu, Yang Liu, Kui Ren, Wenhai Wang, Qing Guo
• Abstract summary: We build an adversarial attack forcing SDEdit to generate a specific data distribution aligned with a specified attribute. We propose the Targeted Attribute Generative Attack (TAGA), using an attribute-aware objective function and optimizing the adversarial noise added to the input stroke painting. Experiments show our method compelling SDEdit to generate a targeted attribute-aware data distribution, significantly outperforming baselines.
• Score: 34.3949228829163
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Guided image synthesis methods, like SDEdit based on the diffusion model, excel at creating realistic images from user inputs such as stroke paintings. However, existing efforts mainly focus on image quality, often overlooking a key point: the diffusion model represents a data distribution, not individual images. This introduces a low but critical chance of generating images that contradict user intentions, raising ethical concerns. For example, a user inputting a stroke painting with female characteristics might, with some probability, get male faces from SDEdit. To expose this potential vulnerability, we aim to build an adversarial attack forcing SDEdit to generate a specific data distribution aligned with a specified attribute (e.g., female), without changing the input's attribute characteristics. We propose the Targeted Attribute Generative Attack (TAGA), using an attribute-aware objective function and optimizing the adversarial noise added to the input stroke painting. Empirical studies reveal that traditional adversarial noise struggles with TAGA, while natural perturbations like exposure and motion blur easily alter generated images' attributes. To execute effective attacks, we introduce FoolSDEdit: We design a joint adversarial exposure and blur attack, adding exposure and motion blur to the stroke painting and optimizing them together. We optimize the execution strategy of various perturbations, framing it as a network architecture search problem. We create the SuperPert, a graph representing diverse execution strategies for different perturbations. After training, we obtain the optimized execution strategy for effective TAGA against SDEdit. Comprehensive experiments on two datasets show our method compelling SDEdit to generate a targeted attribute-aware data distribution, significantly outperforming baselines.

Related papers

• CODE: Confident Ordinary Differential Editing [62.83365660727034]
  Confident Ordinary Differential Editing (CODE) is a novel approach for image synthesis that effectively handles Out-of-Distribution (OoD) guidance images. CODE enhances images through score-based updates along the probability-flow Ordinary Differential Equation (ODE) trajectory. Our method operates in a fully blind manner, relying solely on a pre-trained generative model.
  arXiv  Detail & Related papers  (2024-08-22T14:12:20Z)
• Disrupting Diffusion: Token-Level Attention Erasure Attack against Diffusion-based Customization [19.635385099376066]
  malicious users have misused diffusion-based customization methods like DreamBooth to create fake images. In this paper, we propose DisDiff, a novel adversarial attack method to disrupt the diffusion model outputs.
  arXiv  Detail & Related papers  (2024-05-31T02:45:31Z)
• Unsegment Anything by Simulating Deformation [67.10966838805132]
  "Anything Unsegmentable" is a task to grant any image "the right to be unsegmented" We aim to achieve transferable adversarial attacks against all prompt-based segmentation models. Our approach focuses on disrupting image encoder features to achieve prompt-agnostic attacks.
  arXiv  Detail & Related papers  (2024-04-03T09:09:42Z)
• Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model [61.53213964333474]
  We propose a unified framework Adv-Diffusion that can generate imperceptible adversarial identity perturbations in the latent space but not the raw pixel space. Specifically, we propose the identity-sensitive conditioned diffusion generative model to generate semantic perturbations in the surroundings. The designed adaptive strength-based adversarial perturbation algorithm can ensure both attack transferability and stealthiness.
  arXiv  Detail & Related papers  (2023-12-18T15:25:23Z)
• IMPRESS: Evaluating the Resilience of Imperceptible Perturbations Against Unauthorized Data Usage in Diffusion-Based Generative AI [52.90082445349903]
  Diffusion-based image generation models can create artistic images that mimic the style of an artist or maliciously edit the original images for fake content. Several attempts have been made to protect the original images from such unauthorized data usage by adding imperceptible perturbations. In this work, we introduce a purification perturbation platform, named IMPRESS, to evaluate the effectiveness of imperceptible perturbations as a protective measure.
  arXiv  Detail & Related papers  (2023-10-30T03:33:41Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• TAFIM: Targeted Adversarial Attacks against Facial Image Manipulations [0.0]
  Face image manipulation methods can raise concerns by affecting an individual's privacy or spreading disinformation. In this work, we propose a proactive defense to prevent face manipulation from happening in the first place. We introduce a novel data-driven approach that produces image-specific perturbations which are embedded in the original images.
  arXiv  Detail & Related papers  (2021-12-16T19:00:43Z)
• Generating Image Adversarial Examples by Embedding Digital Watermarks [38.93689142953098]
  We propose a novel digital watermark-based method to generate image adversarial examples to fool deep neural network (DNN) models. We devise an efficient mechanism to select host images and watermark images and utilize the improved discrete wavelet transform (DWT) based watermarking algorithm. Our scheme is able to generate a large number of adversarial examples efficiently, concretely, an average of 1.17 seconds for completing the attacks on each image on the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2020-08-14T09:03:26Z)
• AdvJND: Generating Adversarial Examples with Just Noticeable Difference [3.638233924421642]
  Adding small perturbations on examples causes a good-performance model to misclassify the crafted examples. Adversarial examples generated by our AdvJND algorithm yield distributions similar to those of the original inputs.
  arXiv  Detail & Related papers  (2020-02-01T09:55:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.