SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation

• URL: http://arxiv.org/abs/2505.08665v5
• Date: Fri, 03 Oct 2025 11:48:51 GMT
• Title: SkillFormer: Unified Multi-View Video Understanding for Proficiency Estimation
• Abstract summary: SkillFormer is a parameter-efficient architecture for unified multi-view proficiency estimation.<n>It fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration.<n>It achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency.
• Score: 1.3893859937118993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Assessing human skill levels in complex activities is a challenging problem with applications in sports, rehabilitation, and training. In this work, we present SkillFormer, a parameter-efficient architecture for unified multi-view proficiency estimation from egocentric and exocentric videos. Building on the TimeSformer backbone, SkillFormer introduces a CrossViewFusion module that fuses view-specific features using multi-head cross-attention, learnable gating, and adaptive self-calibration. We leverage Low-Rank Adaptation to fine-tune only a small subset of parameters, significantly reducing training costs. In fact, when evaluated on the EgoExo4D dataset, SkillFormer achieves state-of-the-art accuracy in multi-view settings while demonstrating remarkable computational efficiency, using 4.5x fewer parameters and requiring 3.75x fewer training epochs than prior baselines. It excels in multiple structured tasks, confirming the value of multi-view integration for fine-grained skill assessment. Project page at https://edowhite.github.io/SkillFormer

Related papers

• Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents [9.144810798419975]
  LLM agents increasingly retrieve externally curated skills-procedural instructions retrieved at decision time to improve performance on long-horizon interactive tasks.<n>We propose MASA Model-Aware Skill Alignment, a framework that adapts skills to each target backbone without modifying agent weights.<n> MASA consistently achieves the best overall performance, with gains of up to 25.8 points over the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-29T01:34:42Z)
• Parameter-Efficient Multi-View Proficiency Estimation: From Discriminative Classification to Generative Feedback [1.3893859937118993]
  We discuss three recent contributions to multi-view proficiency estimation on Ego-Exo4D.<n> SkillFormer introduces a parameter-efficient discriminative architecture for selective multi-view fusion.<n> PATS improves temporal sampling by preserving locally dense excerpts of fundamental movements.<n>ProfVLM reformulates proficiency estimation as conditional language generation.
  arXiv  Detail & Related papers  (2026-05-05T15:14:40Z)
• Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinations [5.37133760455631]
  AliAd is a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference.<n>AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility.
  arXiv  Detail & Related papers  (2026-02-09T14:58:08Z)
• MuSASplat: Efficient Sparse-View 3D Gaussian Splats via Lightweight Multi-Scale Adaptation [92.57609195819647]
  MuSASplat is a novel framework that dramatically reduces the computational burden of training pose-free feed-forward 3D Gaussian splats models.<n>Central to our approach is a lightweight Multi-Scale Adapter that enables efficient fine-tuning of ViT-based architectures with only a small fraction of training parameters.
  arXiv  Detail & Related papers  (2025-12-08T04:56:46Z)
• UniRVQA: A Unified Framework for Retrieval-Augmented Vision Question Answering via Self-Reflective Joint Training [16.14877145354785]
  We propose a Unified Retrieval-Augmented VQA framework (UniRVQA) for knowledge-intensive visual questions.<n>UniRVQA adapts general multimodal pre-trained models for fine-grained knowledge-intensive tasks within a unified framework.<n>Our approach achieves competitive performance against state-of-the-art models, delivering a significant 4.7% improvement in answering accuracy, and brings an average 7.5% boost in base MLLMs' VQA performance.
  arXiv  Detail & Related papers  (2025-04-05T05:42:12Z)
• Self-Supervised Partial Cycle-Consistency for Multi-View Matching [5.984724082624813]
  We train a view-invariant feature extraction network with cycle-consistency to handle partial overlap.<n>We present several new cycle variants that complement each other and present a time-divergent scene sampling scheme.<n>Compared to the self-supervised state-of-the-art, we achieve a 4.3 percentage point higher F1 score with our combined contributions.
  arXiv  Detail & Related papers  (2025-01-10T14:32:20Z)
• CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasets [0.9642500063568188]
  This paper presents a new model-agnostic plugin architecture for cross-learning, called CM3T.<n>We introduce two adapter blocks: multi-head vision adapters for transfer learning and cross-attention adapters for multimodal learning.<n>With only 12.8% trainable parameters compared to the backbone to process video input, we achieve comparable and even better results than the state-of-the-art.
  arXiv  Detail & Related papers  (2025-01-06T19:01:10Z)
• Context-Aware Multimodal Pretraining [72.04020920042574]
  We show that vision-language models can be trained to exhibit significantly increased few-shot adaptation. We find up to four-fold improvements in test-time sample efficiency, and average few-shot adaptation gains of over 5%.
  arXiv  Detail & Related papers  (2024-11-22T17:55:39Z)
• CROME: Cross-Modal Adapters for Efficient Multimodal LLM [28.337072921099494]
  Multimodal Large Language Models (MLLMs) demonstrate remarkable image-language capabilities. Existing approaches often necessitate expensive language model retraining and limited adaptability. We propose CROME, an efficient vision-language instruction tuning framework.
  arXiv  Detail & Related papers  (2024-08-13T03:45:11Z)
• Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach [87.8330887605381]
  We show how to adapt a pre-trained Vision Transformer to downstream recognition tasks with only a few learnable parameters. We synthesize a task-specific query with a learnable and lightweight module, which is independent of the pre-trained model. Our method achieves state-of-the-art performance under memory constraints, showcasing its applicability in real-world situations.
  arXiv  Detail & Related papers  (2024-07-09T15:45:04Z)
• VQA Training Sets are Self-play Environments for Generating Few-shot Pools [2.556825820539693]
  We propose a technique in which existing training sets can be directly used for constructing computational environments with task metrics as rewards. The proposed method starts with zero-shot prompts and iteratively refines them by selecting few-shot examples that maximize the task metric on the training set. Our experiments showcase how Gemini learns how to use itself, or another smaller and specialized model such as ScreenAI, to iteratively improve performance on training sets.
  arXiv  Detail & Related papers  (2024-05-30T07:38:58Z)
• Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning [12.5354658533836]
  Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. For artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. We propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches.
  arXiv  Detail & Related papers  (2024-05-06T02:02:57Z)
• Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory [64.11870454160614]
  We propose an efficient Adaptive HOI Detector with Concept-guided Memory (ADA-CM) ADA-CM has two operating modes. The first mode makes it tunable without learning new parameters in a training-free paradigm. Our proposed method achieves competitive results with state-of-the-art on the HICO-DET and V-COCO datasets with much less training time.
  arXiv  Detail & Related papers  (2023-09-07T13:10:06Z)
• Sample Less, Learn More: Efficient Action Recognition via Frame Feature Restoration [59.6021678234829]
  We propose a novel method to restore the intermediate features for two sparsely sampled and adjacent video frames. With the integration of our method, the efficiency of three commonly used baselines has been improved by over 50%, with a mere 0.5% reduction in recognition accuracy.
  arXiv  Detail & Related papers  (2023-07-27T13:52:42Z)
• An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training [79.78201886156513]
  We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Our approach achieves comparable results to single-task state-of-the-art models and demonstrates strong generalization on downstream tasks.
  arXiv  Detail & Related papers  (2023-06-29T17:59:57Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)
• Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes [100.69714600180895]
  offline Q-learning algorithms exhibit strong performance that scales with model capacity. We train a single policy on 40 games with near-human performance using up-to 80 million parameter networks. Compared to return-conditioned supervised approaches, offline Q-learning scales similarly with model capacity and has better performance, especially when the dataset is suboptimal.
  arXiv  Detail & Related papers  (2022-11-28T08:56:42Z)
• Cross-Modal Adapter for Vision-Language Retrieval [60.59577149733934]
  We present a novel Cross-Modal Adapter for parameter-efficient transfer learning.<n>Inspired by adapter-based methods, we adjust the pre-trained model with a few parameterization layers.<n>Our approach has three notable benefits: (1) reduces the vast majority of fine-tuned parameters, (2) saves training time, and (3) allows all the pre-trained parameters to be fixed.
  arXiv  Detail & Related papers  (2022-11-17T16:15:30Z)
• Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Driving [103.745551954983]
  In this paper, we investigate the transfer performance of various types of self-supervised methods, including MoCo and SimCLR, on three downstream tasks. We find that their performances are sub-optimal or even lag far behind the single-task baseline. We propose a simple yet effective pretrain-adapt-finetune paradigm for general multi-task training.
  arXiv  Detail & Related papers  (2022-09-19T12:15:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.