Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample

• URL: http://arxiv.org/abs/2006.12226v3
• Date: Thu, 22 Oct 2020 11:38:19 GMT
• Title: Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample
• Authors: Shir Gur, Sagie Benaim, Lior Wolf
• Abstract summary: We introduce a novel patch-based variational autoencoder (VAE) which allows for a much greater diversity in generation. At coarse scales, our patch-VAE is employed, ensuring samples are of high diversity. At finer scales, a patch-GAN renders the fine details, resulting in high quality videos.
• Score: 107.76407209269236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider the task of generating diverse and novel videos from a single video sample. Recently, new hierarchical patch-GAN based approaches were proposed for generating diverse images, given only a single sample at training time. Moving to videos, these approaches fail to generate diverse samples, and often collapse into generating samples similar to the training video. We introduce a novel patch-based variational autoencoder (VAE) which allows for a much greater diversity in generation. Using this tool, a new hierarchical video generation scheme is constructed: at coarse scales, our patch-VAE is employed, ensuring samples are of high diversity. Subsequently, at finer scales, a patch-GAN renders the fine details, resulting in high quality videos. Our experiments show that the proposed method produces diverse samples in both the image domain, and the more challenging video domain.

Related papers

• One Style is All you Need to Generate a Video [0.9558392439655012]
  We introduce a novel temporal generator based on a set of learned sinusoidal bases. Our method learns dynamic representations of various actions that are independent of image content and can be transferred between different actors.
  arXiv  Detail & Related papers  (2023-10-27T01:17:48Z)
• VIDM: Video Implicit Diffusion Models [75.90225524502759]
  Diffusion models have emerged as a powerful generative method for synthesizing high-quality and diverse set of images. We propose a video generation method based on diffusion models, where the effects of motion are modeled in an implicit condition. We improve the quality of the generated videos by proposing multiple strategies such as sampling space truncation, robustness penalty, and positional group normalization.
  arXiv  Detail & Related papers  (2022-12-01T02:58:46Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• FewGAN: Generating from the Joint Distribution of a Few Images [95.6635227371479]
  We introduce FewGAN, a generative model for generating novel, high-quality and diverse images. FewGAN is a hierarchical patch-GAN that applies quantization at the first coarse scale, followed by a pyramid of residual fully convolutional GANs at finer scales. In an extensive set of experiments, it is shown that FewGAN outperforms baselines both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2022-07-18T07:11:28Z)
• Learning to Generate Novel Scene Compositions from Single Images and Videos [32.131955417610655]
  One-Shot GAN learns to generate samples from a training set as little as one image or one video. We propose a two-branch discriminator, with content and layout branches designed to judge the internal content separately from the scene layout realism.
  arXiv  Detail & Related papers  (2021-05-12T17:59:45Z)
• A Hierarchical Transformation-Discriminating Generative Model for Few Shot Anomaly Detection [93.38607559281601]
  We devise a hierarchical generative model that captures the multi-scale patch distribution of each training image. The anomaly score is obtained by aggregating the patch-based votes of the correct transformation across scales and image regions.
  arXiv  Detail & Related papers  (2021-04-29T17:49:48Z)
• Generating Novel Scene Compositions from Single Images and Videos [21.92417902229955]
  We introduce SIV-GAN, an unconditional generative model that can generate new scene compositions from a single training image or a single video clip. Compared to previous single image GANs, our model generates more diverse, higher quality images, while not being restricted to a single image setting.
  arXiv  Detail & Related papers  (2021-03-24T17:59:07Z)
• Non-Adversarial Video Synthesis with Learned Priors [53.26777815740381]
  We focus on the problem of generating videos from latent noise vectors, without any reference input frames. We develop a novel approach that jointly optimize the input latent space, the weights of a recurrent neural network and a generator through non-adversarial learning. Our approach generates superior quality videos compared to the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-21T02:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.