Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective

• URL: http://arxiv.org/abs/2401.11783v1
• Date: Mon, 22 Jan 2024 09:29:42 GMT
• Title: Full-Body Motion Reconstruction with Sparse Sensing from Graph Perspective
• Authors: Feiyu Yao, Zongkai Wu, Li Yi
• Abstract summary: Estimating 3D full-body pose from sparse sensor data is a pivotal technique employed for the reconstruction of realistic human motions in Augmented Reality and Virtual Reality. We use Body Pose Graph (BPG) to represent the human body and translate the challenge into a prediction problem of graph missing nodes. Our method's effectiveness is evidenced by the attained state-of-the-art performance, particularly in lower body motion, outperforming other baseline methods.
• Score: 22.761692765158646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating 3D full-body pose from sparse sensor data is a pivotal technique employed for the reconstruction of realistic human motions in Augmented Reality and Virtual Reality. However, translating sparse sensor signals into comprehensive human motion remains a challenge since the sparsely distributed sensors in common VR systems fail to capture the motion of full human body. In this paper, we use well-designed Body Pose Graph (BPG) to represent the human body and translate the challenge into a prediction problem of graph missing nodes. Then, we propose a novel full-body motion reconstruction framework based on BPG. To establish BPG, nodes are initially endowed with features extracted from sparse sensor signals. Features from identifiable joint nodes across diverse sensors are amalgamated and processed from both temporal and spatial perspectives. Temporal dynamics are captured using the Temporal Pyramid Structure, while spatial relations in joint movements inform the spatial attributes. The resultant features serve as the foundational elements of the BPG nodes. To further refine the BPG, node features are updated through a graph neural network that incorporates edge reflecting varying joint relations. Our method's effectiveness is evidenced by the attained state-of-the-art performance, particularly in lower body motion, outperforming other baseline methods. Additionally, an ablation study validates the efficacy of each module in our proposed framework.

Related papers

• Occlusion-Aware 3D Motion Interpretation for Abnormal Behavior Detection [10.782354892545651]
  We present OAD2D, which discriminates against motion abnormalities based on reconstructing 3D coordinates of mesh vertices and human joints from monocular videos. We reformulate the abnormal posture estimation by coupling it with Motion to Text (M2T) model in which, the VQVAE is employed to quantize motion features. Our approach demonstrates the robustness of abnormal behavior detection against severe and self-occlusions, as it reconstructs human motion trajectories in global coordinates.
  arXiv  Detail & Related papers  (2024-07-23T18:41:16Z)
• Temporal and Heterogeneous Graph Neural Network for Remaining Useful Life Prediction [27.521188262343596]
  We introduce a novel model named Temporal and Heterogeneous Graph Neural Networks (THGNN) THGNN aggregates historical data from neighboring nodes to accurately capture the temporal dynamics and spatial correlations within the stream of sensor data. We have validated the effectiveness of our approach through comprehensive experiments.
  arXiv  Detail & Related papers  (2024-05-07T14:08:57Z)
• Integrative Deep Learning Framework for Parkinson's Disease Early Detection using Gait Cycle Data Measured by Wearable Sensors: A CNN-GRU-GNN Approach [0.3222802562733786]
  We present a pioneering deep learning architecture tailored for the binary classification of subjects. Our model harnesses the power of 1D-Convolutional Neural Networks (CNN), Gated Recurrent Units (GRU), and Graph Neural Network (GNN) layers. Our proposed model achieves exceptional performance metrics, boasting accuracy, precision, recall, and F1 score values of 99.51%, 99.57%, 99.71%, and 99.64%, respectively.
  arXiv  Detail & Related papers  (2024-04-09T15:19:13Z)
• Dynamic Inertial Poser (DynaIP): Part-Based Motion Dynamics Learning for Enhanced Human Pose Estimation with Sparse Inertial Sensors [17.3834029178939]
  This paper introduces a novel human pose estimation approach using sparse inertial sensors. It leverages a diverse array of real inertial motion capture data from different skeleton formats to improve motion diversity and model generalization. The approach demonstrates superior performance over state-of-the-art models across five public datasets, notably reducing pose error by 19% on the DIP-IMU dataset.
  arXiv  Detail & Related papers  (2023-12-02T13:17:10Z)
• Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering [139.159534903657]
  We develop a generalizable and efficient Neural Radiance Field (NeRF) pipeline for high-fidelity free-viewpoint human body details. To better tackle self-occlusion, we devise a geometry-guided multi-view feature integration approach. For achieving higher rendering efficiency, we introduce a geometry-guided progressive rendering pipeline.
  arXiv  Detail & Related papers  (2021-12-08T14:42:10Z)
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer [55.936527926778695]
  Recent deep learning approaches focus on mining subtle r clues using convolutional neural networks with limited-temporal receptive fields. In this paper, we propose the PhysFormer, an end-to-end video transformer based architecture.
  arXiv  Detail & Related papers  (2021-11-23T18:57:11Z)
• Spatial-Temporal Correlation and Topology Learning for Person Re-Identification in Videos [78.45050529204701]
  We propose a novel framework to pursue discriminative and robust representation by modeling cross-scale spatial-temporal correlation. CTL utilizes a CNN backbone and a key-points estimator to extract semantic local features from human body. It explores a context-reinforced topology to construct multi-scale graphs by considering both global contextual information and physical connections of human body.
  arXiv  Detail & Related papers  (2021-04-15T14:32:12Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• An Adversarial Human Pose Estimation Network Injected with Graph Structure [75.08618278188209]
  In this paper, we design a novel generative adversarial network (GAN) to improve the localization accuracy of visible joints when some joints are invisible. The network consists of two simple but efficient modules, Cascade Feature Network (CFN) and Graph Structure Network (GSN)
  arXiv  Detail & Related papers  (2021-03-29T12:07:08Z)
• Anatomy-aware 3D Human Pose Estimation with Bone-based Pose Decomposition [92.99291528676021]
  Instead of directly regressing the 3D joint locations, we decompose the task into bone direction prediction and bone length prediction. Our motivation is the fact that the bone lengths of a human skeleton remain consistent across time. Our full model outperforms the previous best results on Human3.6M and MPI-INF-3DHP datasets.
  arXiv  Detail & Related papers  (2020-02-24T15:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.