Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs

• URL: http://arxiv.org/abs/2211.14794v1
• Date: Sun, 27 Nov 2022 11:25:35 GMT
• Title: Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs
• Authors: Guangrun Wang, Philip H.S. Torr
• Abstract summary: We show that conventional neural network classifiers can generate high-quality images comparable to state-of-the-art generative models. We propose a mask-based reconstruction module to make semantic gradients-aware to synthesize plausible images. We show that our method is also applicable to text-to-image generation by regarding image-text foundation models.
• Score: 104.72108627191041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Classifiers and generators have long been separated. We break down this separation and showcase that conventional neural network classifiers can generate high-quality images of a large number of categories, being comparable to the state-of-the-art generative models (e.g., DDPMs and GANs). We achieve this by computing the partial derivative of the classification loss function with respect to the input to optimize the input to produce an image. Since it is widely known that directly optimizing the inputs is similar to targeted adversarial attacks incapable of generating human-meaningful images, we propose a mask-based stochastic reconstruction module to make the gradients semantic-aware to synthesize plausible images. We further propose a progressive-resolution technique to guarantee fidelity, which produces photorealistic images. Furthermore, we introduce a distance metric loss and a non-trivial distribution loss to ensure classification neural networks can synthesize diverse and high-fidelity images. Using traditional neural network classifiers, we can generate good-quality images of 256$\times$256 resolution on ImageNet. Intriguingly, our method is also applicable to text-to-image generation by regarding image-text foundation models as generalized classifiers. Proving that classifiers have learned the data distribution and are ready for image generation has far-reaching implications, for classifiers are much easier to train than generative models like DDPMs and GANs. We don't even need to train classification models because tons of public ones are available for download. Also, this holds great potential for the interpretability and robustness of classifiers.

Related papers

• Unlocking Pre-trained Image Backbones for Semantic Image Synthesis [29.688029979801577]
  We propose a new class of GAN discriminators for semantic image synthesis that generates highly realistic images. Our model, which we dub DP-SIMS, achieves state-of-the-art results in terms of image quality and consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes.
  arXiv  Detail & Related papers  (2023-12-20T09:39:19Z)
• Is Deep Learning Network Necessary for Image Generation? [9.131712404284876]
  We investigate the possibility of image generation without using a deep learning network. We validate the assumption that images follow a high-dimensional distribution. Experiments show that our images have a lower FID value compared to those generated by variational auto-encoders.
  arXiv  Detail & Related papers  (2023-08-25T18:14:19Z)
• Your Diffusion Model is Secretly a Zero-Shot Classifier [90.40799216880342]
  We show that density estimates from large-scale text-to-image diffusion models can be leveraged to perform zero-shot classification. Our generative approach to classification attains strong results on a variety of benchmarks. Our results are a step toward using generative over discriminative models for downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T17:59:56Z)
• Revisiting Sparse Convolutional Model for Visual Recognition [40.726494290922204]
  This paper revisits the sparse convolutional modeling for image classification. We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100, and ImageNet datasets.
  arXiv  Detail & Related papers  (2022-10-24T04:29:21Z)
• Meta Internal Learning [88.68276505511922]
  Internal learning for single-image generation is a framework, where a generator is trained to produce novel images based on a single image. We propose a meta-learning approach that enables training over a collection of images, in order to model the internal statistics of the sample image more effectively. Our results show that the models obtained are as suitable as single-image GANs for many common image applications.
  arXiv  Detail & Related papers  (2021-10-06T16:27:38Z)
• Match What Matters: Generative Implicit Feature Replay for Continual Learning [0.0]
  We propose GenIFeR (Generative Implicit Feature Replay) for class-incremental learning. The main idea is to train a generative adversarial network (GAN) to generate images that contain realistic features. We empirically show that GenIFeR is superior to both conventional generative image and feature replay.
  arXiv  Detail & Related papers  (2021-06-09T19:29:41Z)
• Ensembling with Deep Generative Views [72.70801582346344]
  generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars.
  arXiv  Detail & Related papers  (2021-04-29T17:58:35Z)
• Counterfactual Generative Networks [59.080843365828756]
  We propose to decompose the image generation process into independent causal mechanisms that we train without direct supervision. By exploiting appropriate inductive biases, these mechanisms disentangle object shape, object texture, and background. We show that the counterfactual images can improve out-of-distribution with a marginal drop in performance on the original classification task.
  arXiv  Detail & Related papers  (2021-01-15T10:23:12Z)
• Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation [69.10717733870575]
  We present a novel method for guiding generic non-conditional GANs to behave as conditional GANs. Our approach adds into the mix an encoder network to generate the high-dimensional random input that are fed to the generator network of a non-conditional GAN.
  arXiv  Detail & Related papers  (2021-01-04T14:03:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.