Latent Neural Differential Equations for Video Generation

• URL: http://arxiv.org/abs/2011.03864v3
• Date: Tue, 11 May 2021 05:31:13 GMT
• Title: Latent Neural Differential Equations for Video Generation
• Authors: Cade Gordon, Natalie Parde
• Abstract summary: We study the effects of Neural Differential Equations to model the temporal dynamics of video generation. We produce a new state-of-the-art model in 64$times$64 pixel unconditional video generation, with an Inception Score of 15.20.
• Score: 10.127456032874978
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative Adversarial Networks have recently shown promise for video generation, building off of the success of image generation while also addressing a new challenge: time. Although time was analyzed in some early work, the literature has not adequately grown with temporal modeling developments. We study the effects of Neural Differential Equations to model the temporal dynamics of video generation. The paradigm of Neural Differential Equations presents many theoretical strengths including the first continuous representation of time within video generation. In order to address the effects of Neural Differential Equations, we investigate how changes in temporal models affect generated video quality. Our results give support to the usage of Neural Differential Equations as a simple replacement for older temporal generators. While keeping run times similar and decreasing parameter count, we produce a new state-of-the-art model in 64$\times$64 pixel unconditional video generation, with an Inception Score of 15.20.

Related papers

• Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation [5.78796187123888]
  We introduce a method to generate temporally coherent human animation from a single image, a video, or a random noise. We claim that bidirectional temporal modeling enforces temporal coherence on a generative network by largely suppressing the motion ambiguity of human appearance.
  arXiv  Detail & Related papers  (2023-07-02T13:57:45Z)
• Generating Long Videos of Dynamic Scenes [66.56925105992472]
  We present a video generation model that reproduces object motion, changes in camera viewpoint, and new content that arises over time. A common failure case is for content to never change due to over-reliance on inductive biases to provide temporal consistency.
  arXiv  Detail & Related papers  (2022-06-07T16:29:51Z)
• Dynamic Dual-Output Diffusion Models [100.32273175423146]
  Iterative denoising-based generation has been shown to be comparable in quality to other classes of generative models. A major drawback of this method is that it requires hundreds of iterations to produce a competitive result. Recent works have proposed solutions that allow for faster generation with fewer iterations, but the image quality gradually deteriorates.
  arXiv  Detail & Related papers  (2022-03-08T11:20:40Z)
• Physics Informed RNN-DCT Networks for Time-Dependent Partial Differential Equations [62.81701992551728]
  We present a physics-informed framework for solving time-dependent partial differential equations. Our model utilizes discrete cosine transforms to encode spatial and recurrent neural networks. We show experimental results on the Taylor-Green vortex solution to the Navier-Stokes equations.
  arXiv  Detail & Related papers  (2022-02-24T20:46:52Z)
• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks [68.93429034530077]
  We propose dynamics-aware implicit generative adversarial network (DIGAN) for video generation. We show that DIGAN can be trained on 128 frame videos of 128x128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.
  arXiv  Detail & Related papers  (2022-02-21T23:24:01Z)
• Insights from Generative Modeling for Neural Video Compression [31.59496634465347]
  We present newly proposed neural video coding algorithms through the lens of deep autoregressive and latent variable modeling. We propose several architectures that yield state-of-the-art video compression performance on high-resolution video. We provide further evidence that the generative modeling viewpoint can advance the neural video coding field.
  arXiv  Detail & Related papers  (2021-07-28T02:19:39Z)
• Time Dependence in Non-Autonomous Neural ODEs [74.78386661760662]
  We propose a novel family of Neural ODEs with time-varying weights. We outperform previous Neural ODE variants in both speed and representational capacity.
  arXiv  Detail & Related papers  (2020-05-05T01:41:46Z)
• Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation [99.64565200170897]
  We propose a novel human video synthesis method by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. We show several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-01-14T18:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.