AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion
Sensing
- URL: http://arxiv.org/abs/2207.13784v1
- Date: Wed, 27 Jul 2022 20:52:39 GMT
- Title: AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion
Sensing
- Authors: Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich,
Patrick Snape, Christian Holz
- Abstract summary: We present AvatarPoser, the first learning-based method that predicts full-body poses in world coordinates using only motion input from the user's head and hands.
Our method builds on a Transformer encoder to extract deep features from the input signals and decouples global motion from the learned local joint orientations.
In our evaluation, AvatarPoser achieved new state-of-the-art results in evaluations on large motion capture datasets.
- Score: 24.053096294334694
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Today's Mixed Reality head-mounted displays track the user's head pose in
world space as well as the user's hands for interaction in both Augmented
Reality and Virtual Reality scenarios. While this is adequate to support user
input, it unfortunately limits users' virtual representations to just their
upper bodies. Current systems thus resort to floating avatars, whose limitation
is particularly evident in collaborative settings. To estimate full-body poses
from the sparse input sources, prior work has incorporated additional trackers
and sensors at the pelvis or lower body, which increases setup complexity and
limits practical application in mobile settings. In this paper, we present
AvatarPoser, the first learning-based method that predicts full-body poses in
world coordinates using only motion input from the user's head and hands. Our
method builds on a Transformer encoder to extract deep features from the input
signals and decouples global motion from the learned local joint orientations
to guide pose estimation. To obtain accurate full-body motions that resemble
motion capture animations, we refine the arm joints' positions using an
optimization routine with inverse kinematics to match the original tracking
input. In our evaluation, AvatarPoser achieved new state-of-the-art results in
evaluations on large motion capture datasets (AMASS). At the same time, our
method's inference speed supports real-time operation, providing a practical
interface to support holistic avatar control and representation for Metaverse
applications.
Related papers
- Real-Time Simulated Avatar from Head-Mounted Sensors [70.41580295721525]
We present SimXR, a method for controlling a simulated avatar from information (headset pose and cameras) obtained from AR / VR headsets.
To synergize headset poses with cameras, we control a humanoid to track headset movement while analyzing input images to decide body movement.
When body parts are seen, the movements of hands and feet will be guided by the images; when unseen, the laws of physics guide the controller to generate plausible motion.
arXiv Detail & Related papers (2024-03-11T16:15:51Z) - SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data [1.494051815405093]
We introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from sparse data.
Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses.
We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices.
arXiv Detail & Related papers (2023-11-03T18:48:01Z) - Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control.
We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset.
We then create our motion representation by distilling skills directly from the imitator.
arXiv Detail & Related papers (2023-10-06T20:48:43Z) - MOVIN: Real-time Motion Capture using a Single LiDAR [7.3228874258537875]
We present MOVIN, the data-driven generative method for real-time motion capture with global tracking.
Our framework accurately predicts the performer's 3D global information and local joint details.
We implement a real-time application to showcase our method in real-world scenarios.
arXiv Detail & Related papers (2023-09-17T16:04:15Z) - Utilizing Task-Generic Motion Prior to Recover Full-Body Motion from
Very Sparse Signals [3.8079353598215757]
We propose a method that utilizes information from a neural motion prior to improve the accuracy of reconstructed user's motions.
This is based on the premise that the ultimate goal of pose reconstruction is to reconstruct the motion, which is a series of poses.
arXiv Detail & Related papers (2023-08-30T08:21:52Z) - Physics-based Motion Retargeting from Sparse Inputs [73.94570049637717]
Commercial AR/VR products consist only of a headset and controllers, providing very limited sensor data of the user's pose.
We introduce a method to retarget motions in real-time from sparse human sensor data to characters of various morphologies.
We show that the avatar poses often match the user surprisingly well, despite having no sensor information of the lower body available.
arXiv Detail & Related papers (2023-07-04T21:57:05Z) - HOOV: Hand Out-Of-View Tracking for Proprioceptive Interaction using
Inertial Sensing [25.34222794274071]
We present HOOV, a wrist-worn sensing method that allows VR users to interact with objects outside their field of view.
Based on the signals of a single wrist-worn inertial sensor, HOOV continuously estimates the user's hand position in 3-space.
Our novel data-driven method predicts hand positions and trajectories from just the continuous estimation of hand orientation.
arXiv Detail & Related papers (2023-03-13T11:25:32Z) - QuestSim: Human Motion Tracking from Sparse Sensors with Simulated
Avatars [80.05743236282564]
Real-time tracking of human body motion is crucial for immersive experiences in AR/VR.
We present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers.
We show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
arXiv Detail & Related papers (2022-09-20T00:25:54Z) - Drivable Volumetric Avatars using Texel-Aligned Features [52.89305658071045]
Photo telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance.
We propose an end-to-end framework that addresses two core challenges in modeling and driving full-body avatars of real people.
arXiv Detail & Related papers (2022-07-20T09:28:16Z) - Human POSEitioning System (HPS): 3D Human Pose Estimation and
Self-localization in Large Scenes from Body-Mounted Sensors [71.29186299435423]
We introduce (HPS) Human POSEitioning System, a method to recover the full 3D pose of a human registered with a 3D scan of the surrounding environment.
We show that our optimization-based integration exploits the benefits of the two, resulting in pose accuracy free of drift.
HPS could be used for VR/AR applications where humans interact with the scene without requiring direct line of sight with an external camera.
arXiv Detail & Related papers (2021-03-31T17:58:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.