CharacterGAN: Few-Shot Keypoint Character Animation and Reposing

• URL: http://arxiv.org/abs/2102.03141v1
• Date: Fri, 5 Feb 2021 12:38:15 GMT
• Title: CharacterGAN: Few-Shot Keypoint Character Animation and Reposing
• Authors: Tobias Hinz and Matthew Fisher and Oliver Wang and Eli Shechtman and Stefan Wermter
• Abstract summary: We introduce CharacterGAN, a generative model that can be trained on only a few samples of a given character. Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback. We show that our approach outperforms recent baselines and creates realistic animations for diverse characters.
• Score: 64.19520387536741
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We introduce CharacterGAN, a generative model that can be trained on only a few samples (8 - 15) of a given character. Our model generates novel poses based on keypoint locations, which can be modified in real time while providing interactive feedback, allowing for intuitive reposing and animation. Since we only have very limited training samples, one of the key challenges lies in how to address (dis)occlusions, e.g. when a hand moves behind or in front of a body. To address this, we introduce a novel layering approach which explicitly splits the input keypoints into different layers which are processed independently. These layers represent different parts of the character and provide a strong implicit bias that helps to obtain realistic results even with strong (dis)occlusions. To combine the features of individual layers we use an adaptive scaling approach conditioned on all keypoints. Finally, we introduce a mask connectivity constraint to reduce distortion artifacts that occur with extreme out-of-distribution poses at test time. We show that our approach outperforms recent baselines and creates realistic animations for diverse characters. We also show that our model can handle discrete state changes, for example a profile facing left or right, that the different layers do indeed learn features specific for the respective keypoints in those layers, and that our model scales to larger datasets when more data is available.

Related papers

• Purposer: Putting Human Motion Generation in Context [30.706219830149504]
  We present a novel method to generate human motion to populate 3D indoor scenes. It can be controlled with various combinations of conditioning signals such as a path in a scene, target poses, past motions, and scenes represented as 3D point clouds.
  arXiv  Detail & Related papers  (2024-04-19T15:16:04Z)
• DisPositioNet: Disentangled Pose and Identity in Semantic Image Manipulation [83.51882381294357]
  DisPositioNet is a model that learns a disentangled representation for each object for the task of image manipulation using scene graphs. Our framework enables the disentanglement of the variational latent embeddings as well as the feature representation in the graph.
  arXiv  Detail & Related papers  (2022-11-10T11:47:37Z)
• Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing [71.19528222206088]
  We propose a novel Decoupled Multi-task Learning with Cyclical Self-Regulation for face parsing. Specifically, DML-CSR designs a multi-task model which comprises face parsing, binary edge, and category edge detection. Our method achieves the new state-of-the-art performance on the Helen, CelebA-HQ, and LapaMask datasets.
  arXiv  Detail & Related papers  (2022-03-28T02:12:30Z)
• Hierarchical Neural Implicit Pose Network for Animation and Motion Retargeting [66.69067601079706]
  HIPNet is a neural implicit pose network trained on multiple subjects across many poses. We employ a hierarchical skeleton-based representation to learn a signed distance function on a canonical unposed space. We achieve state-of-the-art results on various single-subject and multi-subject benchmarks.
  arXiv  Detail & Related papers  (2021-12-02T03:25:46Z)
• Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression [81.05772887221333]
  We study the dense keypoint regression framework that is previously inferior to the keypoint detection and grouping framework. We present a simple yet effective approach, named disentangled keypoint regression (DEKR) We empirically show that the proposed direct regression method outperforms keypoint detection and grouping methods.
  arXiv  Detail & Related papers  (2021-04-06T05:54:46Z)
• Liquid Warping GAN with Attention: A Unified Framework for Human Image Synthesis [58.05389586712485]
  We tackle human image synthesis, including human motion imitation, appearance transfer, and novel view synthesis. In this paper, we propose a 3D body mesh recovery module to disentangle the pose and shape. We also build a new dataset, namely iPER dataset, for the evaluation of human motion imitation, appearance transfer, and novel view synthesis.
  arXiv  Detail & Related papers  (2020-11-18T02:57:47Z)
• Neural Face Models for Example-Based Visual Speech Synthesis [2.2817442144155207]
  We present a marker-less approach for facial motion capture based on multi-view video. We learn a neural representation of facial expressions, which is used to seamlessly facial performances during the animation procedure.
  arXiv  Detail & Related papers  (2020-09-22T07:35:33Z)
• Efficient Full Image Interactive Segmentation by Leveraging Within-image Appearance Similarity [39.17599924322882]
  We propose a new approach to interactive full-image semantic segmentation. We leverage a key observation: propagation from labeled to unlabeled pixels does not necessarily require class-specific knowledge. We build on this observation and propose an approach capable of jointly propagating pixel labels from multiple classes.
  arXiv  Detail & Related papers  (2020-07-16T08:21:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.