ProSkill: Segment-Level Skill Assessment in Procedural Videos

• URL: http://arxiv.org/abs/2601.20661v1
• Date: Wed, 28 Jan 2026 14:44:09 GMT
• Title: ProSkill: Segment-Level Skill Assessment in Procedural Videos
• Abstract summary: ProSkill is the first benchmark dataset for action-level skill assessment in procedural tasks.<n>We use our dataset to benchmark the main state-of-the-art skill assessment algorithms.
• Score: 25.82207413553515
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Skill assessment in procedural videos is crucial for the objective evaluation of human performance in settings such as manufacturing and procedural daily tasks. Current research on skill assessment has predominantly focused on sports and lacks large-scale datasets for complex procedural activities. Existing studies typically involve only a limited number of actions, focus on either pairwise assessments (e.g., A is better than B) or on binary labels (e.g., good execution vs needs improvement). In response to these shortcomings, we introduce ProSkill, the first benchmark dataset for action-level skill assessment in procedural tasks. ProSkill provides absolute skill assessment annotations, along with pairwise ones. This is enabled by a novel and scalable annotation protocol that allows for the creation of an absolute skill assessment ranking starting from pairwise assessments. This protocol leverages a Swiss Tournament scheme for efficient pairwise comparisons, which are then aggregated into consistent, continuous global scores using an ELO-based rating system. We use our dataset to benchmark the main state-of-the-art skill assessment algorithms, including both ranking-based and pairwise paradigms. The suboptimal results achieved by the current state-of-the-art highlight the challenges and thus the value of ProSkill in the context of skill assessment for procedural videos. All data and code are available at https://fpv-iplab.github.io/ProSkill/

Related papers

• Behavior-Skill: A Fine-Grained Benchmark for Evaluating Vision-Language-Action Policies in Long-Horizon Tasks [39.959375504916494]
  We present Behavior-Skill, a benchmark that reformulates the learning and evaluation of long-horizon tasks.<n>It contains 235,492 skill instances from 10,000 demonstrations across 50 household tasks and 34 semantic skill categories.<n>We show that Behavior-Skill complements full-task evaluation by exposing intermediate capability profiles for analyzing and improving long-horizon VLA policies.
  arXiv  Detail & Related papers  (2026-08-31T10:03:48Z)
• FineSkiing: A Fine-grained Benchmark for Skiing Action Quality Assessment [18.701250140835224]
  Action Quality Assessment (AQA) aims to evaluate and score sports actions.<n>Existing AQA methods primarily predict scores based on features extracted from the entire video.<n>In this paper, we construct the first AQA dataset containing fine-grained sub-score and deduction annotations for aerial skiing.
  arXiv  Detail & Related papers  (2025-11-13T12:29:39Z)
• Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains [97.5573252172065]
  We train a family of Automatic Reasoning Evaluators (FARE) with a simple iterative rejection-sampling supervised finetuning approach.<n>FARE-8B challenges larger specialized RL-trained evaluators and FARE-20B sets the new standard for open-source evaluators.<n>As inference-time rerankers, FARE-20B achieves near-oracle performance on MATH.
  arXiv  Detail & Related papers  (2025-10-20T17:52:06Z)
• Enhancing Classifier Evaluation: A Fairer Benchmarking Strategy Based on Ability and Robustness [0.4749981032986242]
  This study introduces a novel evaluation methodology that combines Item Response Theory (IRT) with the Glicko-2 rating system.<n>IRT assesses classifier ability based on performance over difficult instances, while Glicko-2 updates performance metrics.<n>A case study using the OpenML-CC18 benchmark showed that only 15% of the datasets are truly challenging.
  arXiv  Detail & Related papers  (2025-04-13T23:54:08Z)
• PanguIR Technical Report for NTCIR-18 AEOLLM Task [12.061652026366591]
  Large language models (LLMs) are increasingly critical and challenging to evaluate.<n>Manual evaluation, while comprehensive, is often costly and resource-intensive.<n>automatic evaluation offers greater scalability but is constrained by the limitations of its evaluation criteria.
  arXiv  Detail & Related papers  (2025-03-04T07:40:02Z)
• SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation [75.56845750400116]
  Disaggregated evaluation -- estimation of performance of a machine learning model on different subpopulations -- is a core task when assessing performance and group-fairness of AI systems. We develop SureMap that has high estimation accuracy for both multi-task and single-task disaggregated evaluations of blackbox models. Our method combines maximum a posteriori (MAP) estimation using a well-chosen prior together with cross-validation-free tuning via Stein's unbiased risk estimate (SURE)
  arXiv  Detail & Related papers  (2024-11-14T17:53:35Z)
• CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution [74.41064280094064]
  textbfJudger-1 is the first open-source textbfall-in-one judge LLM. CompassJudger-1 is a general-purpose LLM that demonstrates remarkable versatility. textbfJudgerBench is a new benchmark that encompasses various subjective evaluation tasks.
  arXiv  Detail & Related papers  (2024-10-21T17:56:51Z)
• Self-Training with Pseudo-Label Scorer for Aspect Sentiment Quad Prediction [54.23208041792073]
  Aspect Sentiment Quad Prediction (ASQP) aims to predict all quads (aspect term, aspect category, opinion term, sentiment polarity) for a given review. A key challenge in the ASQP task is the scarcity of labeled data, which limits the performance of existing methods. We propose a self-training framework with a pseudo-label scorer, wherein a scorer assesses the match between reviews and their pseudo-labels.
  arXiv  Detail & Related papers  (2024-06-26T05:30:21Z)
• Hyperparameters in Continual Learning: A Reality Check [50.784080714897776]
  Continual learning (CL) aims to train a model on a sequence of tasks while balancing the trade-off between plasticity (learning new tasks) and stability (retaining prior knowledge)
  arXiv  Detail & Related papers  (2024-03-14T03:13:01Z)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [69.91340332545094]
  We introduce FLASK, a fine-grained evaluation protocol for both human-based and model-based evaluation. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance.
  arXiv  Detail & Related papers  (2023-07-20T14:56:35Z)
• FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding [89.92513889132825]
  We introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-09-27T00:57:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.