Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis

• URL: http://arxiv.org/abs/2111.05916v1
• Date: Wed, 10 Nov 2021 20:18:57 GMT
• Title: Dance In the Wild: Monocular Human Animation with Neural Dynamic Appearance Synthesis
• Authors: Tuanfeng Y. Wang and Duygu Ceylan and Krishna Kumar Singh and Niloy J. Mitra
• Abstract summary: We propose a video based synthesis method that tackles challenges and demonstrates high quality results for in-the-wild videos. We introduce a novel motion signature that is used to modulate the generator weights to capture dynamic appearance changes. We evaluate our method on a set of challenging videos and show that our approach achieves state-of-the art performance both qualitatively and quantitatively.
• Score: 56.550999933048075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Synthesizing dynamic appearances of humans in motion plays a central role in applications such as AR/VR and video editing. While many recent methods have been proposed to tackle this problem, handling loose garments with complex textures and high dynamic motion still remains challenging. In this paper, we propose a video based appearance synthesis method that tackles such challenges and demonstrates high quality results for in-the-wild videos that have not been shown before. Specifically, we adopt a StyleGAN based architecture to the task of person specific video based motion retargeting. We introduce a novel motion signature that is used to modulate the generator weights to capture dynamic appearance changes as well as regularizing the single frame based pose estimates to improve temporal coherency. We evaluate our method on a set of challenging videos and show that our approach achieves state-of-the art performance both qualitatively and quantitatively.

Related papers

• VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models [71.9811050853964]
  VideoJAM is a novel framework that instills an effective motion prior to video generators. VideoJAM achieves state-of-the-art performance in motion coherence. These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation.
  arXiv  Detail & Related papers  (2025-02-04T17:07:10Z)
• A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions [56.709280823844374]
  We introduce a mask-based motion correction module (MCM) that leverages motion context and video mask to repair flawed motions. We also propose a physics-based motion transfer module (PTM), which employs a pretrain and adapt approach for motion imitation. Our approach is designed as a plug-and-play module to physically refine the video motion capture results, including high-difficulty in-the-wild motions.
  arXiv  Detail & Related papers  (2024-12-23T08:26:00Z)
• Dynamic Try-On: Taming Video Virtual Try-on with Dynamic Attention Mechanism [52.9091817868613]
  Video try-on is a promising area for its tremendous real-world potential. Previous research has primarily focused on transferring product clothing images to videos with simple human poses. We propose a novel video try-on framework based on Diffusion Transformer(DiT), named Dynamic Try-On.
  arXiv  Detail & Related papers  (2024-12-13T03:20:53Z)
• Diffusion Priors for Dynamic View Synthesis from Monocular Videos [59.42406064983643]
  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. We first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. We distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields.
  arXiv  Detail & Related papers  (2024-01-10T23:26:41Z)
• MoStGAN-V: Video Generation with Temporal Motion Styles [28.082294960744726]
  Previous works attempt to generate videos in arbitrary lengths either in an autoregressive manner or regarding time as a continuous signal. We argue that a single time-agnostic latent vector of style-based generator is insufficient to model various and temporally-consistent motions. We introduce additional time-dependent motion styles to model diverse motion patterns.
  arXiv  Detail & Related papers  (2023-04-05T22:47:12Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.