Autoencoding Video Latents for Adversarial Video Generation

• URL: http://arxiv.org/abs/2201.06888v1
• Date: Tue, 18 Jan 2022 11:42:14 GMT
• Title: Autoencoding Video Latents for Adversarial Video Generation
• Authors: Sai Hemanth Kasaraneni
• Abstract summary: AVLAE is a two stream latent autoencoder where the video distribution is learned by adversarial training. We demonstrate that our approach learns to disentangle motion and appearance codes even without the explicit structural composition in the generator.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Given the three dimensional complexity of a video signal, training a robust and diverse GAN based video generative model is onerous due to large stochasticity involved in data space. Learning disentangled representations of the data help to improve robustness and provide control in the sampling process. For video generation, there is a recent progress in this area by considering motion and appearance as orthogonal information and designing architectures that efficiently disentangle them. These approaches rely on handcrafting architectures that impose structural priors on the generator to decompose appearance and motion codes in the latent space. Inspired from the recent advancements in the autoencoder based image generation, we present AVLAE (Adversarial Video Latent AutoEncoder) which is a two stream latent autoencoder where the video distribution is learned by adversarial training. In particular, we propose to autoencode the motion and appearance latent vectors of the video generator in the adversarial setting. We demonstrate that our approach learns to disentangle motion and appearance codes even without the explicit structural composition in the generator. Several experiments with qualitative and quantitative results demonstrate the effectiveness of our method.

Related papers

• MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion [3.7270979204213446]
  We present four key contributions to address the challenges of video processing. First, we introduce the 3D Inverted Vector-Quantization Variencoenco Autocoder. Second, we present MotionAura, a text-to-video generation framework. Third, we propose a spectral transformer-based denoising network. Fourth, we introduce a downstream task of Sketch Guided Videopainting.
  arXiv  Detail & Related papers  (2024-10-10T07:07:56Z)
• Video Prediction Models as General Visual Encoders [0.0]
  The researchers propose using video prediction models as general visual encoders, leveraging their ability to capture critical spatial and temporal information. Inspired by human vision studies, the approach aims to develop a latent space representative of motion from images. Experiments involve adapting pre-trained video generative models, analyzing their latent spaces, and training custom decoders for foreground-background segmentation.
  arXiv  Detail & Related papers  (2024-05-25T23:55:47Z)
• MV2MAE: Multi-View Video Masked Autoencoders [33.61642891911761]
  We present a method for self-supervised learning from synchronized multi-view videos. We use a cross-view reconstruction task to inject geometry information in the model. Our approach is based on the masked autoencoder (MAE) framework.
  arXiv  Detail & Related papers  (2024-01-29T05:58:23Z)
• RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks [93.18404922542702]
  We present a novel video generative model designed to address long-term spatial and temporal dependencies. Our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks. Our model synthesizes high-fidelity video clips at a resolution of $256times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps.
  arXiv  Detail & Related papers  (2024-01-11T16:48:44Z)
• StyleInV: A Temporal Style Modulated Inversion Network for Unconditional Video Generation [73.54398908446906]
  We introduce a novel motion generator design that uses a learning-based inversion network for GAN. Our method supports style transfer with simple fine-tuning when the encoder is paired with a pretrained StyleGAN generator.
  arXiv  Detail & Related papers  (2023-08-31T17:59:33Z)
• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks [68.93429034530077]
  We propose dynamics-aware implicit generative adversarial network (DIGAN) for video generation. We show that DIGAN can be trained on 128 frame videos of 128x128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.
  arXiv  Detail & Related papers  (2022-02-21T23:24:01Z)
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis [73.82857768949651]
  We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled.
  arXiv  Detail & Related papers  (2021-04-30T15:38:41Z)
• Non-Adversarial Video Synthesis with Learned Priors [53.26777815740381]
  We focus on the problem of generating videos from latent noise vectors, without any reference input frames. We develop a novel approach that jointly optimize the input latent space, the weights of a recurrent neural network and a generator through non-adversarial learning. Our approach generates superior quality videos compared to the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-21T02:57:33Z)
• An Emerging Coding Paradigm VCM: A Scalable Coding Approach Beyond Feature and Signal [99.49099501559652]
  Video Coding for Machine (VCM) aims to bridge the gap between visual feature compression and classical video coding. We employ a conditional deep generation network to reconstruct video frames with the guidance of learned motion pattern. By learning to extract sparse motion pattern via a predictive model, the network elegantly leverages the feature representation to generate the appearance of to-be-coded frames.
  arXiv  Detail & Related papers  (2020-01-09T14:18:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.