3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data

• URL: http://arxiv.org/abs/2402.13172v4
• Date: Tue, 5 Mar 2024 12:01:35 GMT
• Title: 3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data
• Abstract summary: We propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views. Our experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods.
• Score: 4.130944152992895
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurate 3D kinematics estimation of human body is crucial in various applications for human health and mobility, such as rehabilitation, injury prevention, and diagnosis, as it helps to understand the biomechanical loading experienced during movement. Conventional marker-based motion capture is expensive in terms of financial investment, time, and the expertise required. Moreover, due to the scarcity of datasets with accurate annotations, existing markerless motion capture methods suffer from challenges including unreliable 2D keypoint detection, limited anatomic accuracy, and low generalization capability. In this work, we propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views with consideration of biomechanical prior and spatio-temporal information. To train the model, we create synthetic dataset ODAH with accurate kinematics annotations generated by aligning the body mesh from the SMPL-X model and a full-body OpenSim skeletal model. Our extensive experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods when evaluated across multiple datasets, revealing a promising direction for enhancing video-based human motion capture

Related papers

• Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model [15.358487825950212]
  We extend an existing foundation model, SAM-3D-Body, with an additional biomechanical prediction head that regresses the joint angles and scales of a biomechanical model.<n>To overcome this, we supervise biomechanical outputs with in-loop optimized targets from a Levenberg-Marquardt solver performing inverse kinematics fits against markers from the mesh predictions.<n>We validated this model on biomechanical fits to two marker-based datasets, MoVi and BioCV, as well as movements from a clinical cohort captured with multiview markerless motion capture.
  arXiv  Detail & Related papers  (2026-08-30T17:46:11Z)
• Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements [2.0004822581380775]
  Markerless motion capture (MMC) techniques have been widely beneficial in biomechanical analysis of human movement.<n>This study was to evaluate the performance of a biomechanical reconstruction method with a biomechanical model in the loop for tracking dexterous, unconstrained hand movements.
  arXiv  Detail & Related papers  (2026-07-02T22:14:59Z)
• SOMA: From Surface Observations to Muscle Anatomy [55.23576061150391]
  We present SOMA, a person-specific model that infers bio-temporal muscle behavior from surface signals obtained using RGB cameras.<n>We show how our method provides anatomically grounded animations without the complexity of traditional simulations.
  arXiv  Detail & Related papers  (2026-06-08T09:20:46Z)
• MotionDreamer: Universal Skeletal Motion Generation for 3D Rigged Shapes [52.45449797329548]
  MotionDreamer is a diffusion-based framework for category-agnostic skeletal animation generation from 2D video guidance.<n>We propose a structural-semantic injection mechanism to bridge the gap between 2D visual motion cues and heterogeneous 3D skeletal structures.<n>Our approach significantly outperforms existing methods, setting a new state-of-the-art benchmark for robust and efficient 4D asset generation.
  arXiv  Detail & Related papers  (2026-06-01T00:42:31Z)
• BioHuman: Learning Biomechanical Human Representations from Video [11.653644899302542]
  We introduce a simulation-based framework for estimating muscle activations from existing motion capture datasets.<n>We propose BioHuman, an end-to-end model that takes monocular video as input and jointly predicts human motion and muscle activations.
  arXiv  Detail & Related papers  (2026-05-14T12:36:53Z)
• HumanScore: Benchmarking Human Motions in Generated Videos [16.29197573772623]
  We present HumanScore, a framework to evaluate the quality of human motions in AI-generated videos.<n>HumanScore defines six interpretable metrics spanning kinematic plausibility, temporal stability, and biomechanical consistency.<n>Our analysis reveals consistent gaps between perceptual plausibility and motion biomechanical fidelity.
  arXiv  Detail & Related papers  (2026-04-22T03:51:19Z)
• SIMSPINE: A Biomechanics-Aware Simulation Framework for 3D Spine Motion Annotation and Benchmarking [19.28827026574636]
  We present a biomechanics-aware keypoint simulation framework that augments human pose datasets with anatomically consistent 3D spinal keypoints.<n>We create the first open dataset, named SIMSPINE, which provides sparse vertebra-level 3D spinal annotations for natural full-body motions.<n>With 2.14 million frames, this enables data-driven learning of vertebral kinematics from subtle posture variations.
  arXiv  Detail & Related papers  (2026-02-24T11:31:20Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• MonoMSK: Monocular 3D Musculoskeletal Dynamics Estimation [9.3102957142944]
  We introduce MonoMSK, a hybrid framework that bridges data-driven learning and physics-based simulation.<n>MonoMSK jointly recovers both kinematics (motions) and kinetics (forces and torques) through an anatomically accurate musculoskeletal model.<n> Experiments on BML-MoVi, BEDLAM, and OpenCap show that MonoMSK significantly outperforms state-of-the-art methods in kinematic accuracy.
  arXiv  Detail & Related papers  (2025-11-24T17:20:17Z)
• From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans [50.014530130312714]
  We develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton.<n>We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods.
  arXiv  Detail & Related papers  (2025-09-08T12:24:27Z)
• Validation of Human Pose Estimation and Human Mesh Recovery for Extracting Clinically Relevant Motion Data from Videos [79.62407455005561]
  Marker-less motion capture using human pose estimation produces results in-line with the results of both the IMU and MoCap kinematics. While there is still room for improvement when it comes to the quality of the data produced, we believe that this compromise is within the room of error.
  arXiv  Detail & Related papers  (2025-03-18T22:18:33Z)
• Biomechanics-Guided Residual Approach to Generalizable Human Motion Generation and Estimation [21.750804738752105]
  We propose BioVAE, a biomechanics-aware framework with three core innovations.<n>We show that BioVAE achieves state-of-the-art performance on multiple benchmarks.
  arXiv  Detail & Related papers  (2025-03-08T10:22:36Z)
• BioPose: Biomechanically-accurate 3D Pose Estimation from Monocular Videos [6.280386490530478]
  BioPose is a learning-based framework for predicting biomechanically accurate 3D human pose directly from monocular videos. It includes a Multi-Query Human Mesh Recovery model (MQ-HMR), a Neural Inverse Kinematics (NeurIK) model, and a 2D-informed pose refinement technique. Experiments on benchmark datasets demonstrate that BioPose significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-14T02:56:19Z)
• Learning Speed-Adaptive Walking Agent Using Imitation Learning with Physics-Informed Simulation [0.0]
  We create a skeletal humanoid agent capable of adapting to varying walking speeds while maintaining biomechanically realistic motions. The framework combines a synthetic data generator, which produces biomechanically plausible gait kinematics from open-source biomechanics data, and a training system that uses adversarial imitation learning to train the agent's walking policy.
  arXiv  Detail & Related papers  (2024-12-05T07:55:58Z)
• OpenCap markerless motion capture estimation of lower extremity kinematics and dynamics in cycling [0.0]
  Markerless motion capture offers several benefits over traditional marker-based systems. System can directly detect human body landmarks, reducing manual processing and errors associated with marker placement. This study compares the performance of OpenCap, a markerless motion capture system, with traditional marker-based systems in assessing cycling biomechanics.
  arXiv  Detail & Related papers  (2024-08-20T15:57:40Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• Human Kinematics-inspired Skeleton-based Video Anomaly Detection [3.261881784285304]
  We introduce a new idea called HKVAD (Human Kinematic-inspired Video Anomaly Detection) for video anomaly detection. Our method achieves good results with minimal computational resources, validating its effectiveness and potential.
  arXiv  Detail & Related papers  (2023-09-27T13:52:53Z)
• Unsupervised 3D Pose Estimation with Non-Rigid Structure-from-Motion Modeling [83.76377808476039]
  We propose a new modeling method for human pose deformations and design an accompanying diffusion-based motion prior. Inspired by the field of non-rigid structure-from-motion, we divide the task of reconstructing 3D human skeletons in motion into the estimation of a 3D reference skeleton. A mixed spatial-temporal NRSfMformer is used to simultaneously estimate the 3D reference skeleton and the skeleton deformation of each frame from 2D observations sequence.
  arXiv  Detail & Related papers  (2023-08-18T16:41:57Z)
• Towards Single Camera Human 3D-Kinematics [15.559206592078425]
  We propose a novel approach for direct 3D human kinematic estimation D3KE from videos using deep neural networks. Our experiments demonstrate that the proposed end-to-end training is robust and outperforms 2D and 3D markerless motion capture based kinematic estimation pipelines.
  arXiv  Detail & Related papers  (2023-01-13T08:44:09Z)
• Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening [59.88594294676711]
  Modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions. We propose a system Skeleton2Humanoid'' which performs physics-oriented motion correction at test time. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy.
  arXiv  Detail & Related papers  (2022-10-09T16:15:34Z)
• Differentiable Dynamics for Articulated 3d Human Motion Reconstruction [29.683633237503116]
  We introduce DiffPhy, a differentiable physics-based model for articulated 3d human motion reconstruction from video. We validate the model by demonstrating that it can accurately reconstruct physically plausible 3d human motion from monocular video.
  arXiv  Detail & Related papers  (2022-05-24T17:58:37Z)
• Human Performance Capture from Monocular Video in the Wild [50.34917313325813]
  We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW.
  arXiv  Detail & Related papers  (2021-11-29T16:32:41Z)
• Physics-based Human Motion Estimation and Synthesis from Videos [0.0]
  We propose a framework for training generative models of physically plausible human motion directly from monocular RGB videos. At the core of our method is a novel optimization formulation that corrects imperfect image-based pose estimations. Results show that our physically-corrected motions significantly outperform prior work on pose estimation.
  arXiv  Detail & Related papers  (2021-09-21T01:57:54Z)
• Learning Local Recurrent Models for Human Mesh Recovery [50.85467243778406]
  We present a new method for video mesh recovery that divides the human mesh into several local parts following the standard skeletal model. We then model the dynamics of each local part with separate recurrent models, with each model conditioned appropriately based on the known kinematic structure of the human body. This results in a structure-informed local recurrent learning architecture that can be trained in an end-to-end fashion with available annotations.
  arXiv  Detail & Related papers  (2021-07-27T14:30:33Z)
• Contact and Human Dynamics from Monocular Video [73.47466545178396]
  Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors. We present a physics-based method for inferring 3D human motion from video sequences that takes initial 2D and 3D pose estimates as input.
  arXiv  Detail & Related papers  (2020-07-22T21:09:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.