Assessing Neural Network Robustness via Adversarial Pivotal Tuning

• URL: http://arxiv.org/abs/2211.09782v2
• Date: Sat, 6 Jan 2024 12:51:00 GMT
• Title: Assessing Neural Network Robustness via Adversarial Pivotal Tuning
• Authors: Peter Ebert Christensen, V\'esteinn Sn{\ae}bjarnarson, Andrea Dittadi, Serge Belongie, Sagie Benaim
• Abstract summary: We show how a pretrained image generator can be used to semantically manipulate images in a detailed, diverse, and inversion way. Inspired by recent GAN-based photorealistic methods, we propose a method called Adversarial Pivotal Tuning (APT) We demonstrate that APT is capable of a wide range of class-preserving semantic image manipulations that fool a variety of pretrained classifiers.
• Score: 24.329515700515806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The robustness of image classifiers is essential to their deployment in the real world. The ability to assess this resilience to manipulations or deviations from the training data is thus crucial. These modifications have traditionally consisted of minimal changes that still manage to fool classifiers, and modern approaches are increasingly robust to them. Semantic manipulations that modify elements of an image in meaningful ways have thus gained traction for this purpose. However, they have primarily been limited to style, color, or attribute changes. While expressive, these manipulations do not make use of the full capabilities of a pretrained generative model. In this work, we aim to bridge this gap. We show how a pretrained image generator can be used to semantically manipulate images in a detailed, diverse, and photorealistic way while still preserving the class of the original image. Inspired by recent GAN-based image inversion methods, we propose a method called Adversarial Pivotal Tuning (APT). Given an image, APT first finds a pivot latent space input that reconstructs the image using a pretrained generator. It then adjusts the generator's weights to create small yet semantic manipulations in order to fool a pretrained classifier. APT preserves the full expressive editing capabilities of the generative model. We demonstrate that APT is capable of a wide range of class-preserving semantic image manipulations that fool a variety of pretrained classifiers. Finally, we show that classifiers that are robust to other benchmarks are not robust to APT manipulations and suggest a method to improve them. Code available at: https://captaine.github.io/apt/

Related papers

• Rejuvenating image-GPT as Strong Visual Representation Learners [28.77567067712619]
  This paper enhances image-GPT, one of the pioneering works that introduce autoregressive pretraining to predict the next pixels. We shift the prediction target from raw pixels to semantic tokens, enabling a higher-level understanding of visual content. Experiments showcase that D-iGPT excels as a strong learner of visual representations.
  arXiv  Detail & Related papers  (2023-12-04T18:59:20Z)
• Gradient Adjusting Networks for Domain Inversion [82.72289618025084]
  StyleGAN2 was demonstrated to be a powerful image generation engine that supports semantic editing. We present a per-image optimization method that tunes a StyleGAN2 generator such that it achieves a local edit to the generator's weights. Our experiments show a sizable gap in performance over the current state of the art in this very active domain.
  arXiv  Detail & Related papers  (2023-02-22T14:47:57Z)
• Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models [60.63556257324894]
  A key desired property of image generative models is the ability to disentangle different attributes. We propose a simple, light-weight image editing algorithm where the mixing weights of the two text embeddings are optimized for style matching and content preservation. Experiments show that the proposed method can modify a wide range of attributes, with the performance outperforming diffusion-model-based image-editing algorithms.
  arXiv  Detail & Related papers  (2022-12-16T19:58:52Z)
• Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs [104.72108627191041]
  We show that conventional neural network classifiers can generate high-quality images comparable to state-of-the-art generative models. We propose a mask-based reconstruction module to make semantic gradients-aware to synthesize plausible images. We show that our method is also applicable to text-to-image generation by regarding image-text foundation models.
  arXiv  Detail & Related papers  (2022-11-27T11:25:35Z)
• Masked Autoencoders are Robust Data Augmentors [90.34825840657774]
  Regularization techniques like image augmentation are necessary for deep neural networks to generalize well. We propose a novel perspective of augmentation to regularize the training process. We show that utilizing such model-based nonlinear transformation as data augmentation can improve high-level recognition tasks.
  arXiv  Detail & Related papers  (2022-06-10T02:41:48Z)
• Match What Matters: Generative Implicit Feature Replay for Continual Learning [0.0]
  We propose GenIFeR (Generative Implicit Feature Replay) for class-incremental learning. The main idea is to train a generative adversarial network (GAN) to generate images that contain realistic features. We empirically show that GenIFeR is superior to both conventional generative image and feature replay.
  arXiv  Detail & Related papers  (2021-06-09T19:29:41Z)
• Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation [181.08127307338654]
  This work presents an effective way to exploit the image prior captured by a generative adversarial network (GAN) trained on large-scale natural images. The deep generative prior (DGP) provides compelling results to restore missing semantics, e.g., color, patch, resolution, of various degraded images.
  arXiv  Detail & Related papers  (2020-03-30T17:45:07Z)
• Denoised Smoothing: A Provable Defense for Pretrained Classifiers [101.67773468882903]
  We present a method for provably defending any pretrained image classifier against $ell_p$ adversarial attacks. This method allows public vision API providers and users to seamlessly convert pretrained non-robust classification services into provably robust ones.
  arXiv  Detail & Related papers  (2020-03-04T06:15:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.