Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model

• URL: http://arxiv.org/abs/2608.29928v1
• Date: Sun, 30 Aug 2026 17:46:11 GMT
• Title: Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model
• Abstract summary: We extend an existing foundation model, SAM-3D-Body, with an additional biomechanical prediction head that regresses the joint angles and scales of a biomechanical model.<n>To overcome this, we supervise biomechanical outputs with in-loop optimized targets from a Levenberg-Marquardt solver performing inverse kinematics fits against markers from the mesh predictions.<n>We validated this model on biomechanical fits to two marker-based datasets, MoVi and BioCV, as well as movements from a clinical cohort captured with multiview markerless motion capture.
• Score: 15.358487825950212
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: State-of-the-art monocular body recovery methods predict mesh vertices and angles on the corresponding kinematic tree, but their outputs lack biomechanically defined joint angles that downstream applications like clinical and biomechanical analyses require. We extend an existing foundation model, SAM-3D-Body, with an additional biomechanical prediction head that, from a single RGB image, regresses the joint angles and scales of a biomechanical model. Training this model presents a challenge, as there are limited datasets of paired images and biomechanical fits. To overcome this, we supervise biomechanical outputs with in-loop optimized targets from a Levenberg-Marquardt solver performing inverse kinematics fits against markers from the mesh predictions. This allows distilling the biomechanical head from the mesh head, even from unlabeled images. To make this work with GPU-optimized biomechanical models in MuJoCo, the entire model was implemented in JAX using Equinox. We trained this distilled output head on the publicly released SAM-3D-Body dataset. We then validated this model on biomechanical fits to two publicly available marker-based datasets, MoVi and BioCV, as well as movements from a clinical cohort captured with multiview markerless motion capture. The resulting model outperforms existing models for direct regression of biomechanics from images while only slightly underperforming the state-of-the-art monocular biomechanics method that performs more costly inference-time optimization of entire trajectories.

Related papers

• Pose-to-Biomechanics: Bridging 3D Human Pose Estimation and Biomechanical Attribute Prediction [4.726777092009554]
  BioModule is a lightweight plug-in temporal transformer that attaches downstream of any 3D pose estimator.<n>BioModule is estimator-agnostic and requires no modification of the upstream pose model.<n>We benchmark BioModule across seven state-of-the-art 3D pose estimators.
  arXiv  Detail & Related papers  (2026-07-09T17:31:16Z)
• Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control [86.64250947574749]
  Reconstructing articulated 3D objects is important for animation, gaming, and robotic simulations.<n>Recent neural networks can estimate the articulated structure of 3D objects, but their generalization remains limited by the scarcity of annotated data.<n>We introduce Instruct-Particulate, a model that takes a 3D mesh together with a target kinematic specification.
  arXiv  Detail & Related papers  (2026-06-12T17:59:36Z)
• SOMA: From Surface Observations to Muscle Anatomy [55.23576061150391]
  We present SOMA, a person-specific model that infers bio-temporal muscle behavior from surface signals obtained using RGB cameras.<n>We show how our method provides anatomically grounded animations without the complexity of traditional simulations.
  arXiv  Detail & Related papers  (2026-06-08T09:20:46Z)
• Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models [1.0195618602298682]
  We present a method that combines the SAM 3D Body foundation model with inverse kinematics optimization in a full-body biomechanical model to extract anatomically-constrained finger joint angles from single-view video.<n>This work extends monocular biomechanical analysis to detailed finger tracking, expanding access to quantitative characterization of hand movement from readily available video.
  arXiv  Detail & Related papers  (2026-05-10T02:06:43Z)
• MonoMSK: Monocular 3D Musculoskeletal Dynamics Estimation [9.3102957142944]
  We introduce MonoMSK, a hybrid framework that bridges data-driven learning and physics-based simulation.<n>MonoMSK jointly recovers both kinematics (motions) and kinetics (forces and torques) through an anatomically accurate musculoskeletal model.<n> Experiments on BML-MoVi, BEDLAM, and OpenCap show that MonoMSK significantly outperforms state-of-the-art methods in kinematic accuracy.
  arXiv  Detail & Related papers  (2025-11-24T17:20:17Z)
• From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans [50.014530130312714]
  We develop SKEL, which re-rigs the SMPL body model with a biomechanics skeleton.<n>We show that SKEL has more biomechanically accurate joint locations than SMPL, and the bones fit inside the body surface better than previous methods.
  arXiv  Detail & Related papers  (2025-09-08T12:24:27Z)
• Reconstructing Humans with a Biomechanically Accurate Skeleton [55.06027148976482]
  We introduce a method for reconstructing 3D humans from a single image using a biomechanically accurate skeleton model.<n>Compared to state-of-the-art methods for 3D human mesh recovery, our model achieves competitive performance on standard benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T17:56:24Z)
• Validation of Human Pose Estimation and Human Mesh Recovery for Extracting Clinically Relevant Motion Data from Videos [79.62407455005561]
  Marker-less motion capture using human pose estimation produces results in-line with the results of both the IMU and MoCap kinematics.<n>While there is still room for improvement when it comes to the quality of the data produced, we believe that this compromise is within the room of error.
  arXiv  Detail & Related papers  (2025-03-18T22:18:33Z)
• BioPose: Biomechanically-accurate 3D Pose Estimation from Monocular Videos [6.280386490530478]
  BioPose is a learning-based framework for predicting biomechanically accurate 3D human pose directly from monocular videos.<n>It includes a Multi-Query Human Mesh Recovery model (MQ-HMR), a Neural Inverse Kinematics (NeurIK) model, and a 2D-informed pose refinement technique.<n> Experiments on benchmark datasets demonstrate that BioPose significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-01-14T02:56:19Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• 3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data [4.130944152992895]
  We propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views. Our experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-20T17:33:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.