Correspondence-free online human motion retargeting
- URL: http://arxiv.org/abs/2302.00556v3
- Date: Mon, 4 Mar 2024 10:58:44 GMT
- Title: Correspondence-free online human motion retargeting
- Authors: Rim Rekik, Mathieu Marsot, Anne-H\'el\`ene Olivier, Jean-S\'ebastien
Franco and Stefanie Wuhrer
- Abstract summary: We present a data-driven framework for unsupervised human motion that animates a target subject with the motion of a source subject.
Our method is correspondence-free, requiring neither correspondences between the source and target shapes nor temporal correspondences between different frames of the source motion.
This allows to animate a target shape with arbitrary sequences of humans in motion, possibly captured using 4D acquisition platforms or consumer devices.
- Score: 1.7008985510992145
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a data-driven framework for unsupervised human motion retargeting
that animates a target subject with the motion of a source subject. Our method
is correspondence-free, requiring neither spatial correspondences between the
source and target shapes nor temporal correspondences between different frames
of the source motion. This allows to animate a target shape with arbitrary
sequences of humans in motion, possibly captured using 4D acquisition platforms
or consumer devices. Our method unifies the advantages of two existing lines of
work, namely skeletal motion retargeting, which leverages long-term temporal
context, and surface-based retargeting, which preserves surface details, by
combining a geometry-aware deformation model with a skeleton-aware motion
transfer approach. This allows to take into account long-term temporal context
while accounting for surface details. During inference, our method runs online,
i.e. input can be processed in a serial way, and retargeting is performed in a
single forward pass per frame. Experiments show that including long-term
temporal context during training improves the method's accuracy for skeletal
motion and detail preservation. Furthermore, our method generalizes to
unobserved motions and body shapes. We demonstrate that our method achieves
state-of-the-art results on two test datasets and that it can be used to
animate human models with the output of a multi-view acquisition platform. Code
is available at
\url{https://gitlab.inria.fr/rrekikdi/human-motion-retargeting2023}.
Related papers
- Temporal Residual Jacobians For Rig-free Motion Transfer [45.640576754352104]
We introduce Residual Temporal Jacobians as a novel representation to enable data-driven motion transfer.
Our approach does not assume access to any rigging or intermediate shapes, produces geometrically and temporally consistent motions, and can be used to transfer long motion sequences.
arXiv Detail & Related papers (2024-07-20T18:29:22Z) - Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer [27.278989809466392]
We present a new method for text-driven motion transfer - synthesizing a video that complies with an input text prompt describing the target objects and scene.
We leverage a pre-trained and fixed text-to-video diffusion model, which provides us with generative and motion priors.
arXiv Detail & Related papers (2023-11-28T18:03:27Z) - Task-Oriented Human-Object Interactions Generation with Implicit Neural
Representations [61.659439423703155]
TOHO: Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations.
Our method generates continuous motions that are parameterized only by the temporal coordinate.
This work takes a step further toward general human-scene interaction simulation.
arXiv Detail & Related papers (2023-03-23T09:31:56Z) - Mutual Information-Based Temporal Difference Learning for Human Pose
Estimation in Video [16.32910684198013]
We present a novel multi-frame human pose estimation framework, which employs temporal differences across frames to model dynamic contexts.
To be specific, we design a multi-stage entangled learning sequences conditioned on multi-stage differences to derive informative motion representation sequences.
These place us to rank No.1 in the Crowd Pose Estimation in Complex Events Challenge on benchmark HiEve.
arXiv Detail & Related papers (2023-03-15T09:29:03Z) - Real-time Controllable Motion Transition for Characters [14.88407656218885]
Real-time in-between motion generation is universally required in games and highly desirable in existing animation pipelines.
Our approach consists of two key components: motion manifold and conditional transitioning.
We show that our method is able to generate high-quality motions measured under multiple metrics.
arXiv Detail & Related papers (2022-05-05T10:02:54Z) - MoCaNet: Motion Retargeting in-the-wild via Canonicalization Networks [77.56526918859345]
We present a novel framework that brings the 3D motion task from controlled environments to in-the-wild scenarios.
It is capable of body motion from a character in a 2D monocular video to a 3D character without using any motion capture system or 3D reconstruction procedure.
arXiv Detail & Related papers (2021-12-19T07:52:05Z) - JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion
Retargeting [53.28477676794658]
unsupervised motion in videos has seen substantial advancements through the use of deep neural networks.
We introduce JOKR - a JOint Keypoint Representation that handles both the source and target videos, without requiring any object prior or data collection.
We evaluate our method both qualitatively and quantitatively, and demonstrate that our method handles various cross-domain scenarios, such as different animals, different flowers, and humans.
arXiv Detail & Related papers (2021-06-17T17:32:32Z) - Multi-level Motion Attention for Human Motion Prediction [132.29963836262394]
We study the use of different types of attention, computed at joint, body part, and full pose levels.
Our experiments on Human3.6M, AMASS and 3DPW validate the benefits of our approach for both periodical and non-periodical actions.
arXiv Detail & Related papers (2021-06-17T08:08:11Z) - Hierarchical Style-based Networks for Motion Synthesis [150.226137503563]
We propose a self-supervised method for generating long-range, diverse and plausible behaviors to achieve a specific goal location.
Our proposed method learns to model the motion of human by decomposing a long-range generation task in a hierarchical manner.
On large-scale skeleton dataset, we show that the proposed method is able to synthesise long-range, diverse and plausible motion.
arXiv Detail & Related papers (2020-08-24T02:11:02Z) - History Repeats Itself: Human Motion Prediction via Motion Attention [81.94175022575966]
We introduce an attention-based feed-forward network that explicitly leverages the observation that human motion tends to repeat itself.
In particular, we propose to extract motion attention to capture the similarity between the current motion context and the historical motion sub-sequences.
Our experiments on Human3.6M, AMASS and 3DPW evidence the benefits of our approach for both periodical and non-periodical actions.
arXiv Detail & Related papers (2020-07-23T02:12:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.