Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation

• URL: http://arxiv.org/abs/2402.12676v1
• Date: Tue, 20 Feb 2024 02:48:58 GMT
• Title: Advancing Monocular Video-Based Gait Analysis Using Motion Imitation with Physics-Based Simulation
• Authors: Nikolaos Smyrnakis, Tasos Karakostas, R. James Cotton
• Abstract summary: We use reinforcement learning to control a physics simulation of human movement to replicate the movement seen in video. This forces the inferred movements to be physically plausible, while improving the accuracy of the inferred step length and walking velocity.
• Score: 2.07180164747172
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Gait analysis from videos obtained from a smartphone would open up many clinical opportunities for detecting and quantifying gait impairments. However, existing approaches for estimating gait parameters from videos can produce physically implausible results. To overcome this, we train a policy using reinforcement learning to control a physics simulation of human movement to replicate the movement seen in video. This forces the inferred movements to be physically plausible, while improving the accuracy of the inferred step length and walking velocity.

Related papers

• Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation [54.42523027597904]
  We introduce a novel framework that integrates symbolic regression and trajectory-guided image-to-video (I2V) models for physics-grounded video forecasting.<n>Our approach extracts motion trajectories from input videos, uses a retrieval-based pre-training mechanism to enhance symbolic regression, and discovers equations of motion to forecast physically accurate future trajectories.
  arXiv  Detail & Related papers  (2025-07-09T13:28:42Z)
• KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills [50.34487144149439]
  This paper presents a physics-based humanoid control framework, aiming to master highly-dynamic human behaviors such as Kungfu and dancing.<n>For motion processing, we design a pipeline to extract, filter out, correct, and retarget motions, while ensuring compliance with physical constraints.<n>For motion imitation, we formulate a bi-level optimization problem to dynamically adjust the tracking accuracy tolerance.<n>In experiments, we train whole-body control policies to imitate a set of highly-dynamic motions.
  arXiv  Detail & Related papers  (2025-06-15T13:58:53Z)
• VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior [88.51778468222766]
  Video diffusion models (VDMs) have advanced significantly in recent years, enabling the generation of highly realistic videos. VDMs often fail to produce physically plausible videos due to an inherent lack of understanding of physics. We propose a novel two-stage image-to-video generation framework that explicitly incorporates physics with vision and language informed physical prior.
  arXiv  Detail & Related papers  (2025-03-30T09:03:09Z)
• Spatial-Temporal Graph Diffusion Policy with Kinematic Modeling for Bimanual Robotic Manipulation [88.83749146867665]
  Existing approaches learn a policy to predict a distant next-best end-effector pose. They then compute the corresponding joint rotation angles for motion using inverse kinematics. We propose Kinematics enhanced Spatial-TemporAl gRaph diffuser.
  arXiv  Detail & Related papers  (2025-03-13T17:48:35Z)
• A Plug-and-Play Physical Motion Restoration Approach for In-the-Wild High-Difficulty Motions [56.709280823844374]
  We introduce a mask-based motion correction module (MCM) that leverages motion context and video mask to repair flawed motions. We also propose a physics-based motion transfer module (PTM), which employs a pretrain and adapt approach for motion imitation. Our approach is designed as a plug-and-play module to physically refine the video motion capture results, including high-difficulty in-the-wild motions.
  arXiv  Detail & Related papers  (2024-12-23T08:26:00Z)
• Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos [6.093379844890164]
  We propose a novel method to selectively incorporate the physics models with the kinematics observations in an online setting. A recurrent neural network is introduced to realize a Kalman filter that attentively balances the kinematics input and simulated motion. The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics.
  arXiv  Detail & Related papers  (2024-10-10T10:24:59Z)
• DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with Video Diffusion Priors [75.83647027123119]
  We propose to learn the physical properties of a material field with video diffusion priors. We then utilize a physics-based Material-Point-Method simulator to generate 4D content with realistic motions.
  arXiv  Detail & Related papers  (2024-06-03T16:05:25Z)
• Physics-Guided Human Motion Capture with Pose Probability Modeling [35.159506668475565]
  Existing solutions always adopt kinematic results as reference motions, and the physics is treated as a post-processing module. We employ physics as denoising guidance in the reverse diffusion process to reconstruct human motion from a modeled pose probability distribution. With several iterations, the physics-based tracking and kinematic denoising promote each other to generate a physically plausible human motion.
  arXiv  Detail & Related papers  (2023-08-19T05:28:03Z)
• PhysDiff: Physics-Guided Human Motion Diffusion Model [101.1823574561535]
  Existing motion diffusion models largely disregard the laws of physics in the diffusion process. PhysDiff incorporates physical constraints into the diffusion process. Our approach achieves state-of-the-art motion quality and improves physical plausibility drastically.
  arXiv  Detail & Related papers  (2022-12-05T18:59:52Z)
• Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening [59.88594294676711]
  Modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions. We propose a system Skeleton2Humanoid'' which performs physics-oriented motion correction at test time. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy.
  arXiv  Detail & Related papers  (2022-10-09T16:15:34Z)
• D&D: Learning Human Dynamics from Dynamic Camera [55.60512353465175]
  We present D&D (Learning Human Dynamics from Dynamic Camera), which leverages the laws of physics to reconstruct 3D human motion from the in-the-wild videos with a moving camera. Our approach is entirely neural-based and runs without offline optimization or simulation in physics engines.
  arXiv  Detail & Related papers  (2022-09-19T06:51:02Z)
• Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video [31.96672354594643]
  We focus on the task of estimating a physically plausible articulated human motion from monocular video. Existing approaches that do not consider physics often produce temporally inconsistent output with motion artifacts. We show that our approach achieves competitive results with respect to existing physics-based methods on the Human3.6M benchmark.
  arXiv  Detail & Related papers  (2022-05-24T18:02:49Z)
• Differentiable Dynamics for Articulated 3d Human Motion Reconstruction [29.683633237503116]
  We introduce DiffPhy, a differentiable physics-based model for articulated 3d human motion reconstruction from video. We validate the model by demonstrating that it can accurately reconstruct physically plausible 3d human motion from monocular video.
  arXiv  Detail & Related papers  (2022-05-24T17:58:37Z)
• Physics-based Human Motion Estimation and Synthesis from Videos [0.0]
  We propose a framework for training generative models of physically plausible human motion directly from monocular RGB videos. At the core of our method is a novel optimization formulation that corrects imperfect image-based pose estimations. Results show that our physically-corrected motions significantly outperform prior work on pose estimation.
  arXiv  Detail & Related papers  (2021-09-21T01:57:54Z)
• Contact and Human Dynamics from Monocular Video [73.47466545178396]
  Existing deep models predict 2D and 3D kinematic poses from video that are approximately accurate, but contain visible errors. We present a physics-based method for inferring 3D human motion from video sequences that takes initial 2D and 3D pose estimates as input.
  arXiv  Detail & Related papers  (2020-07-22T21:09:11Z)
• Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects [79.351446087227]
  We address the problem of inferring contact points and the physical forces from videos of humans interacting with objects. Specifically, we use a simulator to predict effects and enforce that estimated forces must lead to the same effect as depicted in the video.
  arXiv  Detail & Related papers  (2020-03-26T17:20:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.