FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment

• URL: http://arxiv.org/abs/2506.03198v3
• Date: Fri, 17 Oct 2025 03:26:07 GMT
• Title: FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment
• Abstract summary: Action Quality Assessment (AQA) has great potential for detecting errors in gym weight training.<n>Existing AQA datasets are limited to single-view competitive sports and RGB video.<n>We introduce FLEX, the first large-scale, multimodal, multiview dataset for fitness AQA.
• Score: 18.287155608368938
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Action Quality Assessment (AQA) -- the task of quantifying how well an action is performed -- has great potential for detecting errors in gym weight training, where accurate feedback is critical to prevent injuries and maximize gains. Existing AQA datasets, however, are limited to single-view competitive sports and RGB video, lacking multimodal signals and professional assessment of fitness actions. We introduce FLEX, the first large-scale, multimodal, multiview dataset for fitness AQA that incorporates surface electromyography (sEMG). FLEX contains over 7,500 multiview recordings of 20 weight-loaded exercises performed by 38 subjects of diverse skill levels, with synchronized RGB video, 3D pose, sEMG, and physiological signals. Expert annotations are organized into a Fitness Knowledge Graph (FKG) linking actions, key steps, error types, and feedback, supporting a compositional scoring function for interpretable quality assessment. FLEX enables multimodal fusion, cross-modal prediction -- including the novel Video$\rightarrow$EMG task -- and biomechanically oriented representation learning. Building on the FKG, we further introduce FLEX-VideoQA, a structured question-answering benchmark with hierarchical queries that drive cross-modal reasoning in vision-language models. Baseline experiments demonstrate that multimodal inputs, multiview video, and fine-grained annotations significantly enhance AQA performance. FLEX thus advances AQA toward richer multimodal settings and provides a foundation for AI-powered fitness assessment and coaching. Dataset and code are available at \href{https://github.com/HaoYin116/FLEX}{https://github.com/HaoYin116/FLEX}. Link to Project \href{https://haoyin116.github.io/FLEX_Dataset}{page}.

Related papers

• MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching [17.624375413094025]
  MyoMechanix is a multimodal ecosystem for weight-loaded actions that aligns motion with muscle activity.<n>It contains 7,500+ samples of 20 actions from 38 subjects, with synchronized multiview RGB video, 3D pose, sEMG, and additional physiological signals.<n> CUBIST performs decomposition-analysis-recomposition for fine-grained error attribution and feedback generation.
  arXiv  Detail & Related papers  (2026-08-26T17:56:33Z)
• Inter-X++: A Comprehensive Benchmark for Multimodal Human-Human Interaction Analysis [55.78822230833247]
  We present Inter-X++, a comprehensive and large-scale benchmark designed to empower versatile HHI analysis.<n>Inter-X++ provides 11,388 high-fidelity interaction sequences and over 8.1M frames, featuring precise whole-body movements and detailed finger articulations.<n>We propose OpenHHI, a single and unified HHI representation and modeling framework that jointly optimize interaction reconstruction and semantic understanding.
  arXiv  Detail & Related papers  (2026-08-20T17:51:48Z)
• HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities [11.096574335599092]
  We introduce HOME-KGQA, a novel KGQA benchmark built on a multimodal KG of daily household activities.<n>Home-KGQA consists of complex, multi-hop natural language questions paired with graph database query languages.<n>Results show that the LLM-based KGQA methods fail to achieve performance comparable to that on existing datasets.
  arXiv  Detail & Related papers  (2026-05-10T06:00:29Z)
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding [98.3098451637867]
  Video-MME-v2 is a comprehensive benchmark designed to rigorously evaluate the robustness and faithfulness of video understanding.<n>To guarantee data quality, Video-MME-v2 is constructed through a rigorously controlled human annotation pipeline.
  arXiv  Detail & Related papers  (2026-04-06T17:59:56Z)
• Learning Perceptual Representations for Gaming NR-VQA with Multi-Task FR Signals [11.586349688474753]
  No-reference video quality assessment (NR-VQA) for gaming videos is challenging due to limited human-rated datasets and unique content characteristics.<n>We present MTL-VQA, a multi-task learning framework that uses full-reference metrics as supervisory signals to learn perceptually meaningful features without human labels for pretraining.
  arXiv  Detail & Related papers  (2026-02-12T12:56:58Z)
• SportR: A Benchmark for Multimodal Large Language Model Reasoning in Sports [21.410115837645318]
  SportR is the first multi-sports large-scale benchmark designed to train and evaluate MLLMs on the fundamental reasoning required for sports intelligence.<n>Our benchmark provides a dataset of 5,017 images and 2,101 videos.<n>For the most advanced tasks requiring multi-step reasoning, such as determining penalties or explaining tactics, we provide 7,118 high-quality, human-authored Chain of Thought annotations.
  arXiv  Detail & Related papers  (2025-11-09T18:55:20Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• CVBench: Evaluating Cross-Video Synergies for Complex Multimodal Understanding and Reasoning [11.478276629279526]
  We present CVBench, the first comprehensive benchmark designed to assess cross-video relational reasoning rigorously.<n>CVBench comprises 1,000 question-answer pairs spanning three tiers: cross-video object association, cross-video event association, and cross-video complex reasoning.<n>Built from five domain-diverse video clusters, the benchmark challenges models to synthesise information across dynamic visual contexts.
  arXiv  Detail & Related papers  (2025-08-27T03:29:35Z)
• RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language [1.5599296461516985]
  RAVEN is a unified QA architecture whose core is QuART, a query-conditioned cross-modal gating module.<n>RAVEN is trained through a three-stage pipeline comprising unimodal pretraining, query-aligned fusion, and disagreement-oriented fine-tuning.<n> Experimental results show RAVEN achieves up to 14.5% and 8.0% gains in accuracy compared to state-of-the-art multi-modal large language models.
  arXiv  Detail & Related papers  (2025-05-21T14:33:36Z)
• Towards Robust Multimodal Physiological Foundation Models: Handling Arbitrary Missing Modalities [9.785262633953794]
  Physio Omni is a foundation model for multimodal physiological signal analysis.<n>It trains a decoupled multimodal tokenizer, enabling masked signal pre-training.<n>It achieves state-of-the-art performance while maintaining strong robustness to missing modalities.
  arXiv  Detail & Related papers  (2025-04-28T09:00:04Z)
• GenM$^3$: Generative Pretrained Multi-path Motion Model for Text Conditional Human Motion Generation [19.2804620329011]
  Generative Pretrained Multi-path Motion Model (GenM$3$) is a framework designed to learn unified motion representations.<n>To enable large-scale training, we integrate and unify 11 high-quality motion datasets.<n>GenM$3$ achieves a state-of-the-art FID of 0.035 on the HumanML3D benchmark, surpassing state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2025-03-19T05:56:52Z)
• M3-AGIQA: Multimodal, Multi-Round, Multi-Aspect AI-Generated Image Quality Assessment [65.3860007085689]
  M3-AGIQA is a comprehensive framework that enables more human-aligned, holistic evaluation of AI-generated images.<n>By aligning model outputs more closely with human judgment, M3-AGIQA delivers robust and interpretable quality scores.
  arXiv  Detail & Related papers  (2025-02-21T03:05:45Z)
• SMPLest-X: Ultimate Scaling for Expressive Human Pose and Shape Estimation [81.36747103102459]
  Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications.<n>Current state-of-the-art methods focus on training innovative architectural designs on confined datasets.<n>We investigate the impact of scaling up EHPS towards a family of generalist foundation models.
  arXiv  Detail & Related papers  (2025-01-16T18:59:46Z)
• Action Quality Assessment via Hierarchical Pose-guided Multi-stage Contrastive Regression [25.657978409890973]
  Action Assessment (AQA) aims at automatic and fair evaluation of athletic performance.<n>Current methods focus on segmenting video into fixed frames, which disrupts the temporal continuity of sub-actions.<n>We propose a novel action quality assessment method through hierarchically pose-guided multi-stage contrastive regression.
  arXiv  Detail & Related papers  (2025-01-07T10:20:16Z)
• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks.<n>Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales.<n>We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• A Framework for Fine-Tuning LLMs using Heterogeneous Feedback [69.51729152929413]
  We present a framework for fine-tuning large language models (LLMs) using heterogeneous feedback. First, we combine the heterogeneous feedback data into a single supervision format, compatible with methods like SFT and RLHF. Next, given this unified feedback dataset, we extract a high-quality and diverse subset to obtain performance increases.
  arXiv  Detail & Related papers  (2024-08-05T23:20:32Z)
• MuJo: Multimodal Joint Feature Space Learning for Human Activity Recognition [2.7532797256542403]
  Human activity recognition (HAR) is a long-standing problem in artificial intelligence with applications in a broad range of areas.<n>We introduce our comprehensive Fitness Multimodal Activity dataset (FiMAD) to enhance HAR performance across various modalities.<n>We show that FiMAD can increase the performance on real HAR datasets such as MM-Fit, MyoGym, MotionSense, and MHEALTH.
  arXiv  Detail & Related papers  (2024-06-06T08:42:36Z)
• Opinion-Unaware Blind Image Quality Assessment using Multi-Scale Deep Feature Statistics [54.08757792080732]
  We propose integrating deep features from pre-trained visual models with a statistical analysis model to achieve opinion-unaware BIQA (OU-BIQA) Our proposed model exhibits superior consistency with human visual perception compared to state-of-the-art BIQA models.
  arXiv  Detail & Related papers  (2024-05-29T06:09:34Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• LOGO: A Long-Form Video Dataset for Group Action Quality Assessment [63.53109605625047]
  We construct a new multi-person long-form video dataset for action quality assessment named LOGO. Our dataset contains 200 videos from 26 artistic swimming events with 8 athletes in each sample along with an average duration of 204.2 seconds. As for richness in annotations, LOGO includes formation labels to depict group information of multiple athletes and detailed annotations on action procedures.
  arXiv  Detail & Related papers  (2024-04-07T17:51:53Z)
• X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning [109.9413329636322]
  This paper introduces an efficient framework that integrates multiple modalities (images, 3D, audio and video) to a frozen Large Language Models (LLMs) Our approach explores two distinct projection mechanisms: Q-Formers and Linear Projections (LPs)
  arXiv  Detail & Related papers  (2023-11-30T18:43:51Z)
• Physics Inspired Hybrid Attention for SAR Target Recognition [61.01086031364307]
  We propose a physics inspired hybrid attention (PIHA) mechanism and the once-for-all (OFA) evaluation protocol to address the issues. PIHA leverages the high-level semantics of physical information to activate and guide the feature group aware of local semantics of target. Our method outperforms other state-of-the-art approaches in 12 test scenarios with same ASC parameters.
  arXiv  Detail & Related papers  (2023-09-27T14:39:41Z)
• Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion [54.33764537135906]
  VideoQA Transformer models demonstrate competitive performance on standard benchmarks. Do these models capture the rich multimodal structures and dynamics from video and text jointly? Are they achieving high scores by exploiting biases and spurious features?
  arXiv  Detail & Related papers  (2023-06-15T06:45:46Z)
• Towards Activated Muscle Group Estimation in the Wild [44.7050439903772]
  We tackle the new task of video-based Activated Muscle Group Estimation (AMGE) aiming at identifying active muscle regions during physical activity in the wild. We provide the MuscleMap dataset featuring >15K video clips with 135 different activities and 20 labeled muscle groups. The proposed MuscleMap dataset is constructed with YouTube videos, specifically targeting High-Intensity Interval Training (HIIT) physical exercise in the wild.
  arXiv  Detail & Related papers  (2023-03-02T04:12:53Z)
• FLAG3D: A 3D Fitness Activity Dataset with Language Instruction [89.60371681477791]
  We present FLAG3D, a large-scale 3D fitness activity dataset with language instruction containing 180K sequences of 60 categories. We show that FLAG3D contributes great research value for various challenges, such as cross-domain human action recognition, dynamic human mesh recovery, and language-guided human action generation.
  arXiv  Detail & Related papers  (2022-12-09T02:33:33Z)
• Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information [77.80071279597665]
  We propose an all-in-one single-stage pre-training approach, named Maximizing Multi-modal Mutual Information Pre-training (M3I Pre-training) Our approach achieves better performance than previous pre-training methods on various vision benchmarks, including ImageNet classification, object detection, LVIS long-tailed object detection, and ADE20k semantic segmentation.
  arXiv  Detail & Related papers  (2022-11-17T18:59:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.