DTSGAN: Learning Dynamic Textures via Spatiotemporal Generative Adversarial Network

• URL: http://arxiv.org/abs/2412.16948v1
• Date: Sun, 22 Dec 2024 09:49:48 GMT
• Title: DTSGAN: Learning Dynamic Textures via Spatiotemporal Generative Adversarial Network
• Authors: Xiangtian Li, Xiaobo Wang, Zhen Qi, Han Cao, Zhaoyang Zhang, Ao Xiang,
• Abstract summary: We introduce atemporal generative adversarial video network (DTSGAN) that can learn from a single dynamic texture. With the pipeline of DTSGAN, a new video sequence is generated from a coarsest scale to the finest one.
• Score: 11.511407106519245
• License:
• Abstract: Dynamic texture synthesis aims to generate sequences that are visually similar to a reference video texture and exhibit specific stationary properties in time. In this paper, we introduce a spatiotemporal generative adversarial network (DTSGAN) that can learn from a single dynamic texture by capturing its motion and content distribution. With the pipeline of DTSGAN, a new video sequence is generated from the coarsest scale to the finest one. To avoid mode collapse, we propose a novel strategy for data updates that helps improve the diversity of generated results. Qualitative and quantitative experiments show that our model is able to generate high quality dynamic textures and natural motion.

Related papers

• Texture Image Synthesis Using Spatial GAN Based on Vision Transformers [1.6482333106552793]
  We propose ViT-SGAN, a new hybrid model that fuses Vision Transformers (ViTs) with a Spatial Generative Adversarial Network (SGAN) to address the limitations of previous methods. By incorporating specialized texture descriptors such as mean-variance (mu, sigma) and textons into the self-attention mechanism of ViTs, our model achieves superior texture synthesis.
  arXiv  Detail & Related papers  (2025-02-03T21:39:30Z)
• 4Dynamic: Text-to-4D Generation with Hybrid Priors [56.918589589853184]
  We propose a novel method for text-to-4D generation, which ensures the dynamic amplitude and authenticity through direct supervision provided by a video prior. Our method not only supports text-to-4D generation but also enables 4D generation from monocular videos.
  arXiv  Detail & Related papers  (2024-07-17T16:02:55Z)
• Learning Dynamic Tetrahedra for High-Quality Talking Head Synthesis [31.90503003079933]
  We introduce Dynamic Tetrahedra (DynTet), a novel hybrid representation that encodes explicit dynamic meshes by neural networks. Compared with prior works, DynTet demonstrates significant improvements in fidelity, lip synchronization, and real-time performance according to various metrics.
  arXiv  Detail & Related papers  (2024-02-27T09:56:15Z)
• Diffusion Priors for Dynamic View Synthesis from Monocular Videos [59.42406064983643]
  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. We first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. We distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields.
  arXiv  Detail & Related papers  (2024-01-10T23:26:41Z)
• Motion-aware Dynamic Graph Neural Network for Video Compressive Sensing [14.67994875448175]
  Video snapshot imaging (SCI) utilizes a 2D detector to capture sequential video frames and compress them into a single measurement. Most existing reconstruction methods are incapable of efficiently capturing long-range spatial and temporal dependencies. We propose a flexible and robust approach based on the graph neural network (GNN) to efficiently model non-local interactions between pixels in space and time regardless of the distance.
  arXiv  Detail & Related papers  (2022-03-01T12:13:46Z)
• Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks [68.93429034530077]
  We propose dynamics-aware implicit generative adversarial network (DIGAN) for video generation. We show that DIGAN can be trained on 128 frame videos of 128x128 resolution, 80 frames longer than the 48 frames of the previous state-of-the-art method.
  arXiv  Detail & Related papers  (2022-02-21T23:24:01Z)
• Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis [56.550999933048075]
  We propose a video based synthesis method that tackles challenges and demonstrates high quality results for in-the-wild videos. We introduce a novel motion signature that is used to modulate the generator weights to capture dynamic appearance changes. We evaluate our method on a set of challenging videos and show that our approach achieves state-of-the art performance both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2021-11-10T20:18:57Z)
• Dynamic View Synthesis from Dynamic Monocular Video [69.80425724448344]
  We present an algorithm for generating views at arbitrary viewpoints and any input time step given a monocular video of a dynamic scene. We show extensive quantitative and qualitative results of dynamic view synthesis from casually captured videos.
  arXiv  Detail & Related papers  (2021-05-13T17:59:50Z)
• Dynamic Texture Synthesis by Incorporating Long-range Spatial and Temporal Correlations [27.247382497265214]
  We introduce a new loss term, called the Shifted Gram loss, to capture the structural and long-range correlation of the reference texture video. We also introduce a frame sampling strategy to exploit long-period motion across multiple frames.
  arXiv  Detail & Related papers  (2021-04-13T05:04:51Z)
• Neural Human Video Rendering by Learning Dynamic Textures and Rendering-to-Video Translation [99.64565200170897]
  We propose a novel human video synthesis method by explicitly disentangling the learning of time-coherent fine-scale details from the embedding of the human in 2D screen space. We show several applications of our approach, such as human reenactment and novel view synthesis from monocular video, where we show significant improvement over the state of the art both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-01-14T18:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.