MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints

• URL: http://arxiv.org/abs/2404.10227v1
• Date: Tue, 16 Apr 2024 02:18:18 GMT
• Title: MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints
• Authors: Pengfei Xie, Wenqiang Xu, Tutian Tang, Zhenjun Yu, Cewu Lu,
• Abstract summary: We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
• Score: 50.61346764110482
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This work proposes a novel learning framework for visual hand dynamics analysis that takes into account the physiological aspects of hand motion. The existing models, which are simplified joint-actuated systems, often produce unnatural motions. To address this, we integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create a new model, MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We further propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron (MLP) network. Our evaluation of the accuracy of MS-MANO and the efficacy of the BioPR is conducted in two separate parts. The accuracy of MS-MANO is compared with MyoSuite, while the efficacy of BioPR is benchmarked against two large-scale public datasets and two recent state-of-the-art methods. The results demonstrate that our approach consistently improves the baseline methods both quantitatively and qualitatively.

Related papers

• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• 3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data [4.130944152992895]
  We propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views. Our experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-20T17:33:40Z)
• A Multi-Grained Symmetric Differential Equation Model for Learning Protein-Ligand Binding Dynamics [74.93549765488103]
  In drug discovery, molecular dynamics simulation provides a powerful tool for predicting binding affinities, estimating transport properties, and exploring pocket sites. We propose NeuralMD, the first machine learning surrogate that can facilitate numerical MD and provide accurate simulations in protein-ligand binding. We show the efficiency and effectiveness of NeuralMD, with a 2000$times$ speedup over standard numerical MD simulation and outperforming all other ML approaches by up to 80% under the stability metric.
  arXiv  Detail & Related papers  (2024-01-26T09:35:17Z)
• Micro-Macro Consistency in Multiscale Modeling: Score-Based Model Assisted Sampling of Fast/Slow Dynamical Systems [0.0]
  In the study of physics-based multi-time-scale dynamical systems, techniques have been developed for enhancing sampling. In the field of Machine Learning, a generic goal of generative models is to sample from a target density, after training on empirical samples from this density. In this work, we show that that SGMs can be used in such a coupling framework to improve sampling in multiscale dynamical systems.
  arXiv  Detail & Related papers  (2023-12-10T00:46:37Z)
• A Physics-Informed Low-Shot Learning For sEMG-Based Estimation of Muscle Force and Joint Kinematics [4.878073267556235]
  Muscle force and joint kinematics estimation from surface electromyography (sEMG) are essential for real-time biomechanical analysis. Recent advances in deep neural networks (DNNs) have shown the potential to improve biomechanical analysis in a fully automated and reproducible manner. This paper presents a novel physics-informed low-shot learning method for sEMG-based estimation of muscle force and joint kinematics.
  arXiv  Detail & Related papers  (2023-07-08T23:01:12Z)
• Extending Process Discovery with Model Complexity Optimization and Cyclic States Identification: Application to Healthcare Processes [62.997667081978825]
  The paper presents an approach to process mining providing semi-automatic support to model optimization. A model simplification approach is proposed, which essentially abstracts the raw model at the desired granularity. We aim to demonstrate the capabilities of the technological solution using three datasets from different applications in the healthcare domain.
  arXiv  Detail & Related papers  (2022-06-10T16:20:59Z)
• End-to-End Learning of Hybrid Inverse Dynamics Models for Precise and Compliant Impedance Control [16.88250694156719]
  We present a novel hybrid model formulation that enables us to identify fully physically consistent inertial parameters of a rigid body dynamics model. We compare our approach against state-of-the-art inverse dynamics models on a 7 degree of freedom manipulator.
  arXiv  Detail & Related papers  (2022-05-27T07:39:28Z)
• Learning Local Recurrent Models for Human Mesh Recovery [50.85467243778406]
  We present a new method for video mesh recovery that divides the human mesh into several local parts following the standard skeletal model. We then model the dynamics of each local part with separate recurrent models, with each model conditioned appropriately based on the known kinematic structure of the human body. This results in a structure-informed local recurrent learning architecture that can be trained in an end-to-end fashion with available annotations.
  arXiv  Detail & Related papers  (2021-07-27T14:30:33Z)
• Energy-based View of Retrosynthesis [70.66156081030766]
  We propose a framework that unifies sequence- and graph-based methods as energy-based models. We present a novel dual variant within the framework that performs consistent training over Bayesian forward- and backward-prediction. This model improves state-of-the-art performance by 9.6% for template-free approaches where the reaction type is unknown.
  arXiv  Detail & Related papers  (2020-07-14T18:51:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.