From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching

• URL: http://arxiv.org/abs/2603.26938v1
• Date: Fri, 27 Mar 2026 19:26:28 GMT
• Title: From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching
• Abstract summary: BioCoach is a vision-language framework for fitness coaching from streaming video.<n>It fuses visual appearance and 3D skeletal kinematics, through a novel three-stage pipeline.<n>On the original QEVD-fit-coach, it improves text quality and correctness with near-parity timing.
• Score: 19.008659008661542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present BioCoach, a biomechanics-grounded vision--language framework for fitness coaching from streaming video. BioCoach fuses visual appearance and 3D skeletal kinematics, through a novel three-stage pipeline: an exercise-specific degree-of-freedom selector that focuses analysis on salient joints; a structured biomechanical context that pairs individualized morphometrics with cycle and constraint analysis; and a vision--biomechanics conditioned feedback module that applies cross-attention to generate precise, actionable text. Using parameter-efficient training that freezes the vision and language backbones, BioCoach yields transparent, personalized reasoning rather than pattern matching. To enable learning and fair evaluation, we augment QEVD-fit-coach with biomechanics-oriented feedback to create QEVD-bio-fit-coach, and we introduce a biomechanics-aware LLM judge metric. BioCoach delivers clear gains on QEVD-bio-fit-coach across lexical and judgment metrics while maintaining temporal triggering; on the original QEVD-fit-coach, it improves text quality and correctness with near-parity timing, demonstrating that explicit kinematics and constraints are key to accurate, phase-aware coaching.

Related papers

• Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model [15.358487825950212]
  We extend an existing foundation model, SAM-3D-Body, with an additional biomechanical prediction head that regresses the joint angles and scales of a biomechanical model.<n>To overcome this, we supervise biomechanical outputs with in-loop optimized targets from a Levenberg-Marquardt solver performing inverse kinematics fits against markers from the mesh predictions.<n>We validated this model on biomechanical fits to two marker-based datasets, MoVi and BioCV, as well as movements from a clinical cohort captured with multiview markerless motion capture.
  arXiv  Detail & Related papers  (2026-08-30T17:46:11Z)
• MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching [17.624375413094025]
  MyoMechanix is a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity.<n>It contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals.<n> CUBIST performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation.
  arXiv  Detail & Related papers  (2026-08-26T17:56:33Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• MyoChallenge 2025: A New Benchmark for Human Athletic Intelligence [57.70335388371609]
  MyoChallenge at NeurIPS 2025 established a pioneering benchmark for motor control intelligence in sports.<n>High-fidelity musculoskeletal models within physics simulation combined with machine learning-driven algorithms.<n>Challenge attracted almost 70 teams and over 560 submissions globally.
  arXiv  Detail & Related papers  (2026-05-15T06:09:44Z)
• PhyCo: Learning Controllable Physical Priors for Generative Motion [55.59209981836171]
  We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation.<n>Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback.
  arXiv  Detail & Related papers  (2026-04-30T17:53:03Z)
• BioGait-VLM: A Tri-Modal Vision-Language-Biomechanics Framework for Interpretable Clinical Gait Assessment [4.674098893032597]
  BioGait-VLM is a tri-modal Vision-Language-Biomechanics framework for interpretable clinical gait assessment.<n>Our architecture incorporates a Temporal Evidence Distillation branch to capture rhythmic dynamics and a Biomechanical Tokenization branch that projects 3D skeleton sequences into language-aligned semantic tokens.
  arXiv  Detail & Related papers  (2026-03-09T16:25:28Z)
• Exploring Physical Intelligence Emergence via Omni-Modal Architecture and Physical Data Engine [50.62040226184694]
  We present OmniFysics, a compact omni-modal model that unifies understanding across images, audio, video, and text.<n>To inject explicit physical knowledge, we build a physical data engine with two components.<n>Experiments show competitive performance on standard multimodal benchmarks and improved results on physics-oriented evaluations.
  arXiv  Detail & Related papers  (2026-02-05T14:04:51Z)
• Talking Tennis: Language Feedback from 3D Biomechanical Action Recognition [0.0]
  This research project develops a novel framework that extracts key biomechanical features from motion data.<n>These features are analyzed for relationships influencing stroke effectiveness and injury risk, forming the basis for feedback generation.<n>The experimental setup evaluates this framework on classification performance and interpretability, bridging the gap between explainable AI and sports biomechanics.
  arXiv  Detail & Related papers  (2025-10-04T19:55:30Z)
• Biomechanics-Guided Residual Approach to Generalizable Human Motion Generation and Estimation [21.750804738752105]
  We propose BioVAE, a biomechanics-aware framework with three core innovations.<n>We show that BioVAE achieves state-of-the-art performance on multiple benchmarks.
  arXiv  Detail & Related papers  (2025-03-08T10:22:36Z)
• MS-MANO: Enabling Hand Pose Tracking with Biomechanical Constraints [50.61346764110482]
  We integrate a musculoskeletal system with a learnable parametric hand model, MANO, to create MS-MANO. This model emulates the dynamics of muscles and tendons to drive the skeletal system, imposing physiologically realistic constraints on the resulting torque trajectories. We also propose a simulation-in-the-loop pose refinement framework, BioPR, that refines the initial estimated pose through a multi-layer perceptron network.
  arXiv  Detail & Related papers  (2024-04-16T02:18:18Z)
• 3D Kinematics Estimation from Video with a Biomechanical Model and Synthetic Training Data [4.130944152992895]
  We propose a novel biomechanics-aware network that directly outputs 3D kinematics from two input views. Our experiments demonstrate that the proposed approach, only trained on synthetic data, outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-02-20T17:33:40Z)
• xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein [74.64101864289572]
  We propose a unified protein language model, xTrimoPGLM, to address protein understanding and generation tasks simultaneously.<n>xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories.<n>It can also generate de novo protein sequences following the principles of natural ones, and can perform programmable generation after supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-01-11T15:03:17Z)
• Attentive Symmetric Autoencoder for Brain MRI Segmentation [56.02577247523737]
  We propose a novel Attentive Symmetric Auto-encoder based on Vision Transformer (ViT) for 3D brain MRI segmentation tasks. In the pre-training stage, the proposed auto-encoder pays more attention to reconstruct the informative patches according to the gradient metrics. Experimental results show that our proposed attentive symmetric auto-encoder outperforms the state-of-the-art self-supervised learning methods and medical image segmentation models.
  arXiv  Detail & Related papers  (2022-09-19T09:43:19Z)
• SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory [61.44510300515693]
  We study the task of simultaneous lip and devise SimulLR, a simultaneous lip Reading transducer with attention-guided adaptive memory. The experiments show that the SimulLR achieves the translation speedup 9.10 times times compared with the state-of-the-art non-simultaneous methods.
  arXiv  Detail & Related papers  (2021-08-31T05:54:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.