SAM4Dcap: Training-free Biomechanical Twin System from Monocular Video

• URL: http://arxiv.org/abs/2602.13760v1
• Date: Sat, 14 Feb 2026 13:09:38 GMT
• Title: SAM4Dcap: Training-free Biomechanical Twin System from Monocular Video
• Abstract summary: SAM4Dcap is an open-source framework for estimating biomechanical metrics from monocular video without additional training.<n>The pipeline converts reconstructed meshes into trajectory files compatible with diverse musculoskeletal models.<n>Preliminary evaluations on walking and drop-jump tasks indicate that SAM4Dcap has the potential to achieve knee kinematic predictions comparable to multi-view systems.
• Score: 16.202462062701038
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Quantitative biomechanical analysis is essential for clinical diagnosis and injury prevention but is often restricted to laboratories due to the high cost of optical motion capture systems. While multi-view video approaches have lowered barriers, they remain impractical for home-based scenarios requiring monocular capture. This paper presents SAM4Dcap, an open-source, end-to-end framework for estimating biomechanical metrics from monocular video without additional training. SAM4Dcap integrates the temporally consistent 4D human mesh recovery of SAM-Body4D with the OpenSim biomechanical solver. The pipeline converts reconstructed meshes into trajectory files compatible with diverse musculoskeletal models. We introduce automated prompting strategies and a Linux-native build for processing. Preliminary evaluations on walking and drop-jump tasks indicate that SAM4Dcap has the potential to achieve knee kinematic predictions comparable to multi-view systems, although some discrepancies in hip flexion and residual jitter remain. By bridging advanced computer vision with established biomechanical simulation, SAM4Dcap provides a flexible, accessible foundation for non-laboratory motion analysis.

Related papers

• Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements [2.0004822581380775]
  Markerless motion capture (MMC) techniques have been widely beneficial in biomechanical analysis of human movement.<n>This study was to evaluate the performance of a biomechanical reconstruction method with a biomechanical model in the loop for tracking dexterous, unconstrained hand movements.
  arXiv  Detail & Related papers  (2026-07-02T22:14:59Z)
• From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video [0.6768558752130311]
  I present a physics-free pipeline to predict instantaneous 3D hip and knee contact forces from an uncalibrated monocular video.<n>Parametric body meshes are recovered per frame, encoded as kinematic features, and decoded into forces by a transformer.<n>This pipeline establishes uncalibrated monocular video as a viable modality for estimating joint loading.
  arXiv  Detail & Related papers  (2026-06-04T18:31:00Z)
• A Digital Twin Framework for Virtual Visuo-Haptic Teleoperation of Complex-Shaped Optical Microrobots [6.975054201075641]
  This paper presents a digital twin framework for virtual visuo-haptic teleoperation of complex-shaped optical microrobots.<n>The framework integrates a digital twin environment, image-based pose and depth estimation, microrobot motion simulation, and model-based haptic rendering.<n>Results show the framework's effectiveness for evaluating visuo-haptic teleoperation strategies for complex-shaped optical microrobots.
  arXiv  Detail & Related papers  (2026-05-27T13:15:38Z)
• Scaling Video Pretraining for Surgical Foundation Models [51.92777479821822]
  SurgRec is a scalable and reproducible pretraining recipe for surgical video understanding.<n>We curate a large corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
  arXiv  Detail & Related papers  (2026-03-31T16:31:25Z)
• OpenCap Monocular: 3D Human Kinematics and Musculoskeletal Dynamics from a Single Smartphone Video [0.2012293145190529]
  Quantifying human movement (kinematics) and musculoskeletal forces (kinetics) at scale could transform prediction, treatment, and monitoring of mobility-related conditions.<n>We introduce OpenCap Monocular, an algorithm that estimates 3D skeletal kinematics and kinetics from a single smartphone video.
  arXiv  Detail & Related papers  (2026-03-25T18:56:46Z)
• Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation [63.04810454548667]
  We introduce PhysGen, a scalable continuous and sequential world interaction framework to solve robotic manipulation tasks.<n>By treating the pretrained video model as a proxy for a physics simulator, PhysGen models the dynamic interplay between the external environment and robot actions.<n>We introduce a multimodal continuous representation that unifies video and action into shared physical tokens, bridging the gap between discrete video generation and continuous robotic control.
  arXiv  Detail & Related papers  (2026-02-18T14:58:18Z)
• MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery [4.287077788799387]
  MiDAS is a platform-agnostic system for time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms.<n>We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of peg transfer and hernia repair suturing tasks.
  arXiv  Detail & Related papers  (2026-02-12T20:56:15Z)
• SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos [53.227781131348856]
  Human Mesh Recovery aims to reconstruct 3D human pose and shape from 2D observations.<n>Recent image-based HMR methods such as SAM 3D Body achieve strong robustness on in-the-wild images.<n>We propose SAM-Body4D, a training-free framework for temporally consistent and occlusion-robust HMR from videos.
  arXiv  Detail & Related papers  (2025-12-09T09:37:31Z)
• MonoMSK: Monocular 3D Musculoskeletal Dynamics Estimation [9.3102957142944]
  We introduce MonoMSK, a hybrid framework that bridges data-driven learning and physics-based simulation.<n>MonoMSK jointly recovers both kinematics (motions) and kinetics (forces and torques) through an anatomically accurate musculoskeletal model.<n> Experiments on BML-MoVi, BEDLAM, and OpenCap show that MonoMSK significantly outperforms state-of-the-art methods in kinematic accuracy.
  arXiv  Detail & Related papers  (2025-11-24T17:20:17Z)
• Is Single-View Mesh Reconstruction Ready for Robotics? [78.14584238127338]
  We evaluate single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation.<n>Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.
  arXiv  Detail & Related papers  (2025-05-23T14:35:56Z)
• BioPose: Biomechanically-accurate 3D Pose Estimation from Monocular Videos [6.280386490530478]
  BioPose is a learning-based framework for predicting biomechanically accurate 3D human pose directly from monocular videos.<n>It includes a Multi-Query Human Mesh Recovery model (MQ-HMR), a Neural Inverse Kinematics (NeurIK) model, and a 2D-informed pose refinement technique.<n> Experiments on benchmark datasets demonstrate that BioPose significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-14T02:56:19Z)
• Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction [51.49400490437258]
  This work develops a method for imitating articulated object manipulation from a single monocular RGB human demonstration. We first propose 4D Differentiable Part Models (4D-DPM), a method for recovering 3D part motion from a monocular video. Given this 4D reconstruction, the robot replicates object trajectories by planning bimanual arm motions that induce the demonstrated object part motion. We evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part trajectories and RSRD's physical execution performance on 9 objects across 10 trials each on a bimanual YuMi robot.
  arXiv  Detail & Related papers  (2024-09-26T17:57:16Z)
• Comparison of Kinematics and Kinetics Between OpenCap and a Marker-Based Motion Capture System in Cycling [0.0]
  This study evaluates the agreement of marker-based and markerless (OpenCap) motion capture systems in assessing joint kinematics and kinetics during cycling.<n>Key metrics, including joint angles, moments, and joint reaction loads, were computed using OpenSim.<n>Results revealed very strong agreement (r GT 0.9) for hip (flexion/extension), knee (flexion/extension), and ankle (dorsiflexion/plantarflexion) joint angles.
  arXiv  Detail & Related papers  (2024-08-20T15:57:40Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videos [79.50191812646125]
  Reconstruction of endoscopic scenes is an important asset for various medical applications, from post-surgery analysis to educational training. We adress the challenging setup of a moving endoscope within a highly dynamic environment of deforming tissue. We propose an implicit scene separation into multiple overlapping 4D neural radiance fields (NeRFs) and a progressive optimization scheme jointly optimizing for reconstruction and camera poses from scratch. This improves the ease-of-use and allows to scale reconstruction capabilities in time to process surgical videos of 5,000 frames and more; an improvement of more than ten times compared to the state of the art while being agnostic to external tracking information
  arXiv  Detail & Related papers  (2024-03-18T19:13:02Z)
• 3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data [4.130944152992895]
  We propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views. Our experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-20T17:33:40Z)
• Robotic Navigation Autonomy for Subretinal Injection via Intelligent Real-Time Virtual iOCT Volume Slicing [88.99939660183881]
  We propose a framework for autonomous robotic navigation for subretinal injection. Our method consists of an instrument pose estimation method, an online registration between the robotic and the i OCT system, and trajectory planning tailored for navigation to an injection target. Our experiments on ex-vivo porcine eyes demonstrate the precision and repeatability of the method.
  arXiv  Detail & Related papers  (2023-01-17T21:41:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.