HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions

• URL: http://arxiv.org/abs/2203.09287v1
• Date: Thu, 17 Mar 2022 12:30:17 GMT
• Title: HybridCap: Inertia-aid Monocular Capture of Challenging Human Motions
• Authors: Han Liang, Yannan He, Chengfeng Zhao, Mutian Li, Jingya Wang, Jingyi Yu, Lan Xu
• Abstract summary: We present a light-weight, hybrid mocap technique called HybridCap. It augments the camera with only 4 Inertial Measurement Units (IMUs) in a learning-and-optimization framework. It can robustly handle challenging movements ranging from fitness actions to Latin dance.
• Score: 41.56735523771541
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular 3D motion capture (mocap) is beneficial to many applications. The use of a single camera, however, often fails to handle occlusions of different body parts and hence it is limited to capture relatively simple movements. We present a light-weight, hybrid mocap technique called HybridCap that augments the camera with only 4 Inertial Measurement Units (IMUs) in a learning-and-optimization framework. We first employ a weakly-supervised and hierarchical motion inference module based on cooperative Gated Recurrent Unit (GRU) blocks that serve as limb, body and root trackers as well as an inverse kinematics solver. Our network effectively narrows the search space of plausible motions via coarse-to-fine pose estimation and manages to tackle challenging movements with high efficiency. We further develop a hybrid optimization scheme that combines inertial feedback and visual cues to improve tracking accuracy. Extensive experiments on various datasets demonstrate HybridCap can robustly handle challenging movements ranging from fitness actions to Latin dance. It also achieves real-time performance up to 60 fps with state-of-the-art accuracy.

Related papers

• COIN: Control-Inpainting Diffusion Prior for Human and Camera Motion Estimation [98.05046790227561]
  COIN is a control-inpainting motion diffusion prior that enables fine-grained control to disentangle human and camera motions. COIN outperforms the state-of-the-art methods in terms of global human motion estimation and camera motion estimation.
  arXiv  Detail & Related papers  (2024-08-29T10:36:29Z)
• Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring [71.60457491155451]
  Eliminating image blur produced by various kinds of motion has been a challenging problem. We propose a novel real-world deblurring filtering model called the Motion-adaptive Separable Collaborative Filter. Our method provides an effective solution for real-world motion blur removal and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-19T19:44:24Z)
• Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera [10.055317239956423]
  We present a lightweight and affordable motion capture method based on two smartwatches and a head-mounted camera. Our method can make wearable motion capture accessible to everyone everywhere, enabling 3D full-body motion capture in diverse environments.
  arXiv  Detail & Related papers  (2024-01-01T18:56:54Z)
• Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
  We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator.
  arXiv  Detail & Related papers  (2023-10-06T20:48:43Z)
• Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model [18.139630622759636]
  We present AGRoL, a novel conditional diffusion model specifically designed to track full bodies given sparse upper-body tracking signals. Our model is based on a simple multi-layer perceptron (MLP) architecture and a novel conditioning scheme for motion data. Unlike common diffusion architectures, our compact architecture can run in real-time, making it suitable for online body-tracking applications.
  arXiv  Detail & Related papers  (2023-04-17T19:35:13Z)
• QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars [80.05743236282564]
  Real-time tracking of human body motion is crucial for immersive experiences in AR/VR. We present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers. We show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
  arXiv  Detail & Related papers  (2022-09-20T00:25:54Z)
• Neural Monocular 3D Human Motion Capture with Physical Awareness [76.55971509794598]
  We present a new trainable system for physically plausible markerless 3D human motion capture. Unlike most neural methods for human motion capture, our approach is aware of physical and environmental constraints. It produces smooth and physically principled 3D motions in an interactive frame rate in a wide variety of challenging scenes.
  arXiv  Detail & Related papers  (2021-05-03T17:57:07Z)
• SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videos [40.19723456533343]
  We propose SportsCap -- the first approach for simultaneously capturing 3D human motions and understanding fine-grained actions from monocular challenging sports video input. Our approach utilizes the semantic and temporally structured sub-motion prior in the embedding space for motion capture and understanding. Based on such hybrid motion information, we introduce a multi-stream spatial-temporal Graph Convolutional Network(ST-GCN) to predict the fine-grained semantic action attributes.
  arXiv  Detail & Related papers  (2021-04-23T07:52:03Z)
• ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal References [18.327101908143113]
  We propose ChallenCap -- a template-based approach to capture challenging 3D human motions using a single RGB camera. We adopt a novel learning-and-optimization framework, with the aid of multi-modal references. Experiments on our new challenging motion dataset demonstrate the effectiveness and robustness of our approach to capture challenging human motions.
  arXiv  Detail & Related papers  (2021-03-11T15:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.