GH-Feat: Learning Versatile Generative Hierarchical Features from GANs
- URL: http://arxiv.org/abs/2301.05315v1
- Date: Thu, 12 Jan 2023 21:59:46 GMT
- Title: GH-Feat: Learning Versatile Generative Hierarchical Features from GANs
- Authors: Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou
- Abstract summary: We show that a generative feature learned from image synthesis exhibits great potentials in solving a wide range of computer vision tasks.
We first train an encoder by considering the pretrained StyleGAN generator as a learned loss function.
The visual features produced by our encoder, termed as Generative Hierarchical Features (GH-Feat), highly align with the layer-wise GAN representations.
- Score: 61.208757845344074
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent years witness the tremendous success of generative adversarial
networks (GANs) in synthesizing photo-realistic images. GAN generator learns to
compose realistic images and reproduce the real data distribution. Through
that, a hierarchical visual feature with multi-level semantics spontaneously
emerges. In this work we investigate that such a generative feature learned
from image synthesis exhibits great potentials in solving a wide range of
computer vision tasks, including both generative ones and more importantly
discriminative ones. We first train an encoder by considering the pretrained
StyleGAN generator as a learned loss function. The visual features produced by
our encoder, termed as Generative Hierarchical Features (GH-Feat), highly align
with the layer-wise GAN representations, and hence describe the input image
adequately from the reconstruction perspective. Extensive experiments support
the versatile transferability of GH-Feat across a range of applications, such
as image editing, image processing, image harmonization, face verification,
landmark detection, layout prediction, image retrieval, etc. We further show
that, through a proper spatial expansion, our developed GH-Feat can also
facilitate fine-grained semantic segmentation using only a few annotations.
Both qualitative and quantitative results demonstrate the appealing performance
of GH-Feat.
Related papers
- Rethinking the Up-Sampling Operations in CNN-based Generative Network
for Generalizable Deepfake Detection [86.97062579515833]
We introduce the concept of Neighboring Pixel Relationships(NPR) as a means to capture and characterize the generalized structural artifacts stemming from up-sampling operations.
A comprehensive analysis is conducted on an open-world dataset, comprising samples generated by tft28 distinct generative models.
This analysis culminates in the establishment of a novel state-of-the-art performance, showcasing a remarkable tft11.6% improvement over existing methods.
arXiv Detail & Related papers (2023-12-16T14:27:06Z) - In-Domain GAN Inversion for Faithful Reconstruction and Editability [132.68255553099834]
We propose in-domain GAN inversion, which consists of a domain-guided domain-regularized and a encoder to regularize the inverted code in the native latent space of the pre-trained GAN model.
We make comprehensive analyses on the effects of the encoder structure, the starting inversion point, as well as the inversion parameter space, and observe the trade-off between the reconstruction quality and the editing property.
arXiv Detail & Related papers (2023-09-25T08:42:06Z) - Distilling Representations from GAN Generator via Squeeze and Span [55.76208869775715]
We propose to distill knowledge from GAN generators by squeezing and spanning their representations.
We span the distilled representation of the synthetic domain to the real domain by also using real training data to remedy the mode collapse of GANs.
arXiv Detail & Related papers (2022-11-06T01:10:28Z) - Joint Learning of Deep Texture and High-Frequency Features for
Computer-Generated Image Detection [24.098604827919203]
We propose a joint learning strategy with deep texture and high-frequency features for CG image detection.
A semantic segmentation map is generated to guide the affine transformation operation.
The combination of the original image and the high-frequency components of the original and rendered images are fed into a multi-branch neural network equipped with attention mechanisms.
arXiv Detail & Related papers (2022-09-07T17:30:40Z) - A Method for Evaluating Deep Generative Models of Images via Assessing
the Reproduction of High-order Spatial Context [9.00018232117916]
Generative adversarial networks (GANs) are one kind of DGM which are widely employed.
In this work, we demonstrate several objective tests of images output by two popular GAN architectures.
We designed several context models (SCMs) of distinct image features that can be recovered after generation by a trained GAN.
arXiv Detail & Related papers (2021-11-24T15:58:10Z) - Ensembling with Deep Generative Views [72.70801582346344]
generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose.
Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification.
We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars.
arXiv Detail & Related papers (2021-04-29T17:58:35Z) - Detection, Attribution and Localization of GAN Generated Images [24.430919035100317]
We propose a novel approach to detect, attribute and localize GAN generated images.
A deep learning network is then trained on these features to detect, attribute and localize these GAN generated/manipulated images.
A large scale evaluation of our approach on 5 GAN datasets shows promising results in detecting GAN generated images.
arXiv Detail & Related papers (2020-07-20T20:49:34Z) - Generative Hierarchical Features from Synthesizing Images [65.66756821069124]
We show that learning to synthesize images can bring remarkable hierarchical visual features that are generalizable across a wide range of applications.
The visual feature produced by our encoder, termed as Generative Hierarchical Feature (GH-Feat), has strong transferability to both generative and discriminative tasks.
arXiv Detail & Related papers (2020-07-20T18:04:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.