FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2608.08736v1
• Date: Sun, 09 Aug 2026 14:28:17 GMT
• Title: FitAQA: A Benchmark of Fitness Action Quality Assessment for Multimodal Large Language Models
• Abstract summary: We introduce FitAQA, a systematic benchmark for evaluating Multimodal Large Language Models (MLLMs) in fitness AQA.<n>We develop a unified form error taxonomy that defines 38 recurring form errors within six complementary quality dimensions.<n> FitAQA further formulates three evaluation tasks: perception for recognizing relevant visual evidence, judgement for combining that evidence with domain knowledge to assess execution correctness, and temporal grounding for localizing form errors over time.
• Score: 12.824373478041737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fitness Action Quality Assessment (AQA) is important for intelligent sports training, yet the capabilities of Multimodal Large Language Models (MLLMs) in this setting remain underexplored. Existing benchmarks rely on action-specific annotation schemes and focus primarily on final assessment outputs, offering limited insight into how models assess exercise quality. We introduce FitAQA, a systematic benchmark for evaluating MLLMs in fitness AQA, containing 2,219 videos and 5,512 QA instances across 30 bodyweight exercises. In collaboration with experts in sports science, we develop a unified form error taxonomy that defines 38 recurring form errors within six complementary quality dimensions: alignment, symmetry, stability, coordination, tempo, and completeness. This taxonomy provides a shared assessment framework across different exercises. FitAQA further formulates three evaluation tasks: perception for recognizing relevant visual evidence, judgement for combining that evidence with domain knowledge to assess execution correctness, and temporal grounding for localizing form errors over time. Extensive evaluation shows that current MLLMs still struggle to assess exercise quality comprehensively and localize form errors precisely. Controlled experiments further indicate that visual perception is a key bottleneck, as judgement performance improves substantially when ground-truth perceptual evidence is provided. The dataset and evaluation code will be made publicly available.

Related papers

• PointQ-Bench: Benchmarking Diagnostic and Interpretable Point Cloud Quality Assessment [28.88904276309088]
  Point cloud quality plays a critical role in 3D acquisition, reconstruction, rendering, and perception.<n>We introduce PointQ-Bench, a benchmark designed to extend PCQA from scalar scoring toward comprehensive quality understanding.<n>PointQ-Bench consists of 3,083 point clouds spanning authentic scans, simulated distortions, and AI-generated content, covering eight major issue types.
  arXiv  Detail & Related papers  (2026-05-27T09:56:36Z)
• ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning [59.558706734431276]
  Current evaluations of spatial intelligence can be systematically invalid under modern vision-language model (VLM) settings.<n>We introduce ReVSI, a benchmark and protocol that ensures each QA pair is answerable and correct under the model's actual inputs.
  arXiv  Detail & Related papers  (2026-04-27T10:45:51Z)
• Can Vision Language Models Judge Action Quality? An Empirical Evaluation [5.688409551177917]
  Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging.<n>Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised.<n>We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies.
  arXiv  Detail & Related papers  (2026-04-09T14:29:19Z)
• Decoupling Perception and Calibration: Label-Efficient Image Quality Assessment Framework [78.58395822978271]
  LEAF is a Label-Efficient Image Quality Assessment Framework.<n>It distills perceptual quality priors from an MLLM teacher into a lightweight student regressor.<n>Our method significantly reduces the need for human annotations while maintaining strong MOS-aligned correlations.
  arXiv  Detail & Related papers  (2026-01-28T15:15:17Z)
• Beyond Description: Cognitively Benchmarking Fine-Grained Action for Embodied Agents [52.14392337070763]
  We introduce CFG-Bench, a new benchmark designed to systematically evaluate fine-grained action intelligence.<n>CFG-Bench consists of 1,368 curated videos paired with 19,562 three-modalities question-answer pairs targeting four cognitive abilities.<n>Our comprehensive evaluation on CFG-Bench reveals that leading MLLMs struggle to produce detailed instructions for physical interactions.
  arXiv  Detail & Related papers  (2025-11-24T02:02:29Z)
• Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization [53.82400605816587]
  Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.<n>A major challenge lies in the non-stationary nature of quality distributions in real-world scenarios.<n>We introduce Continual AQA (CAQA), which equips with Continual Learning capabilities to handle evolving distributions.
  arXiv  Detail & Related papers  (2025-10-08T10:09:47Z)
• Evaluating Variance in Visual Question Answering Benchmarks [0.9065034043031668]
  Multimodal large language models (MLLMs) have emerged as powerful tools for visual question answering (VQA)<n>Despite their advancements, the evaluation of MLLMs on VQA benchmarks often relies on point estimates.<n>This paper critically examines these issues by analyzing across 14 widely used VQA benchmarks.
  arXiv  Detail & Related papers  (2025-08-04T17:37:13Z)
• Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens [30.083110119139793]
  We introduce textbfAbilityLens, a unified benchmark designed to evaluate MLLMs in six key perception abilities.<n>We identify the strengths and weaknesses of current main-stream MLLMs, highlighting stability patterns and revealing a notable performance gap between state-of-the-art open-source and closed-source models.
  arXiv  Detail & Related papers  (2024-11-22T04:41:20Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels [95.44077384918725]
  We propose to teach large multi-modality models (LMMs) with text-defined rating levels instead of scores. The proposed Q-Align achieves state-of-the-art performance on image quality assessment (IQA), image aesthetic assessment (IAA) and video quality assessment (VQA) tasks.
  arXiv  Detail & Related papers  (2023-12-28T16:10:25Z)
• Uncertainty-Driven Action Quality Assessment [11.958132175629368]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores.<n>We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss.<n>Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Evaluation Gaps in Machine Learning Practice [13.963766987258161]
  In practice, evaluations of machine learning models frequently focus on a narrow range of decontextualized predictive behaviours. We examine the evaluation gaps between the idealized breadth of evaluation concerns and the observed narrow focus of actual evaluations. By studying these properties, we demonstrate the machine learning discipline's implicit assumption of a range of commitments which have normative impacts.
  arXiv  Detail & Related papers  (2022-05-11T04:00:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.