Going beyond Free Viewpoint: Creating Animatable Volumetric Video of
Human Performances
- URL: http://arxiv.org/abs/2009.00922v1
- Date: Wed, 2 Sep 2020 09:46:12 GMT
- Title: Going beyond Free Viewpoint: Creating Animatable Volumetric Video of
Human Performances
- Authors: Anna Hilsmann, Philipp Fechteler, Wieland Morgenstern, Wolfgang Paier,
Ingo Feldmann, Oliver Schreer, Peter Eisert
- Abstract summary: We present an end-to-end pipeline for the creation of high-quality animatable volumetric video content of human performances.
Semantic enrichment and geometric animation ability are achieved by establishing temporal consistency in the 3D data.
For pose editing, we exploit the captured data as much as possible and kinematically deform the captured frames to fit a desired pose.
- Score: 7.7824496657259665
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we present an end-to-end pipeline for the creation of
high-quality animatable volumetric video content of human performances. Going
beyond the application of free-viewpoint volumetric video, we allow
re-animation and alteration of an actor's performance through (i) the
enrichment of the captured data with semantics and animation properties and
(ii) applying hybrid geometry- and video-based animation methods that allow a
direct animation of the high-quality data itself instead of creating an
animatable model that resembles the captured data. Semantic enrichment and
geometric animation ability are achieved by establishing temporal consistency
in the 3D data, followed by an automatic rigging of each frame using a
parametric shape-adaptive full human body model. Our hybrid geometry- and
video-based animation approaches combine the flexibility of classical CG
animation with the realism of real captured data. For pose editing, we exploit
the captured data as much as possible and kinematically deform the captured
frames to fit a desired pose. Further, we treat the face differently from the
body in a hybrid geometry- and video-based animation approach where coarse
movements and poses are modeled in the geometry only, while very fine and
subtle details in the face, often lacking in purely geometric methods, are
captured in video-based textures. These are processed to be interactively
combined to form new facial expressions. On top of that, we learn the
appearance of regions that are challenging to synthesize, such as the teeth or
the eyes, and fill in missing regions realistically in an autoencoder-based
approach. This paper covers the full pipeline from capturing and producing
high-quality video content, over the enrichment with semantics and deformation
properties for re-animation and processing of the data for the final hybrid
animation.
Related papers
- PhysAnimator: Physics-Guided Generative Cartoon Animation [19.124321553546242]
We introduce PhysAnimator, a novel approach for generating anime-stylized animation from static anime illustrations.
To capture the fluidity and exaggeration characteristic of anime, we perform image-space deformable body simulations on extracted mesh geometries.
We extract and warp sketches from the simulation sequence, generating a texture-agnostic representation, and employ a sketch-guided video diffusion model to synthesize high-quality animation frames.
arXiv Detail & Related papers (2025-01-27T22:48:36Z) - CFSynthesis: Controllable and Free-view 3D Human Video Synthesis [57.561237409603066]
CFSynthesis is a novel framework for generating high-quality human videos with customizable attributes.
Our method leverages a texture-SMPL-based representation to ensure consistent and stable character appearances across free viewpoints.
Results on multiple datasets show that CFSynthesis achieves state-of-the-art performance in complex human animations.
arXiv Detail & Related papers (2024-12-15T05:57:36Z) - Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenes [49.26872036160368]
We propose a method for animating parts of high-quality 3D scenes in a Gaussian Splatting representation.
We find that, in contrast to prior work, this enables realistic animations of complex, pre-existing 3D scenes.
arXiv Detail & Related papers (2024-11-28T16:01:58Z) - Make-It-Animatable: An Efficient Framework for Authoring Animation-Ready 3D Characters [86.13319549186959]
We present Make-It-Animatable, a novel data-driven method to make any 3D humanoid model ready for character animation in less than one second.
Our framework generates high-quality blend weights, bones, and pose transformations.
Compared to existing methods, our approach demonstrates significant improvements in both quality and speed.
arXiv Detail & Related papers (2024-11-27T10:18:06Z) - Alignment is All You Need: A Training-free Augmentation Strategy for Pose-guided Video Generation [19.408715783816167]
We introduce a training-free framework that ensures the generated video sequence preserves the reference image's subtleties.
We decouple skeletal and motion priors from pose information, enabling precise control over animation generation.
Our method significantly enhances the quality of video generation without the need for large datasets or expensive computational resources.
arXiv Detail & Related papers (2024-08-29T13:08:12Z) - Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation [27.700371215886683]
diffusion models have become the mainstream in visual generation research, owing to their robust generative capabilities.
In this paper, we propose a novel framework tailored for character animation.
By expanding the training data, our approach can animate arbitrary characters, yielding superior results in character animation compared to other image-to-video methods.
arXiv Detail & Related papers (2023-11-28T12:27:15Z) - MagicAnimate: Temporally Consistent Human Image Animation using
Diffusion Model [74.84435399451573]
This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence.
Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion.
We introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity.
arXiv Detail & Related papers (2023-11-27T18:32:31Z) - Unsupervised Learning of Style-Aware Facial Animation from Real Acting
Performances [3.95944314850151]
We present a novel approach for text/speech-driven animation of a photo-realistic head model based on blend-shape geometry, dynamic textures, and neural rendering.
Our animation method is based on a conditional CNN that transforms text or speech into a sequence of animation parameters.
For realistic real-time rendering, we train a U-Net that refines pixelization-based renderings by computing improved colors and a foreground matte.
arXiv Detail & Related papers (2023-06-16T17:58:04Z) - Render In-between: Motion Guided Video Synthesis for Action
Interpolation [53.43607872972194]
We propose a motion-guided frame-upsampling framework that is capable of producing realistic human motion and appearance.
A novel motion model is trained to inference the non-linear skeletal motion between frames by leveraging a large-scale motion-capture dataset.
Our pipeline only requires low-frame-rate videos and unpaired human motion data but does not require high-frame-rate videos for training.
arXiv Detail & Related papers (2021-11-01T15:32:51Z) - Deep Animation Video Interpolation in the Wild [115.24454577119432]
In this work, we formally define and study the animation video code problem for the first time.
We propose an effective framework, AnimeInterp, with two dedicated modules in a coarse-to-fine manner.
Notably, AnimeInterp shows favorable perceptual quality and robustness for animation scenarios in the wild.
arXiv Detail & Related papers (2021-04-06T13:26:49Z) - A Robust Interactive Facial Animation Editing System [0.0]
We propose a new learning-based approach to easily edit a facial animation from a set of intuitive control parameters.
We use a resolution-preserving fully convolutional neural network that maps control parameters to blendshapes coefficients sequences.
The proposed system is robust and can handle coarse, exaggerated edits from non-specialist users.
arXiv Detail & Related papers (2020-07-18T08:31:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.