Towards Universal Skeleton-Based Action Recognition

• URL: http://arxiv.org/abs/2604.17013v1
• Date: Sat, 18 Apr 2026 14:50:46 GMT
• Title: Towards Universal Skeleton-Based Action Recognition
• Abstract summary: This work studies the problem of heterogeneous skeleton-based action recognition with open vocabularies.<n>We propose a Transformer-based model that comprises three key components: unified skeleton representation, motion encoder for skeletons, and multi-grained motion-text alignment.<n>Experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the ability of the proposed method.
• Score: 26.447920160010515
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the development of robotics, skeleton-based action recognition has become increasingly important, as human-robot interaction requires understanding the actions of humans and humanoid robots. Due to different sources of human skeletons and structures of humanoid robots, skeleton data naturally exhibit heterogeneity. However, previous works overlook the data heterogeneity of skeletons and solely construct models using homogeneous skeletons. Moreover, open-vocabulary action recognition is also essential for real-world applications. To this end, this work studies the challenging problem of heterogeneous skeleton-based action recognition with open vocabularies. We construct a large-scale Heterogeneous Open-Vocabulary (HOV) Skeleton dataset by integrating and refining multiple representative large-scale skeleton-based action datasets. To address universal skeleton-based action recognition, we propose a Transformer-based model that comprises three key components: unified skeleton representation, motion encoder for skeletons, and multi-grained motion-text alignment. The motion encoder feeds multi-modal skeleton embeddings into a two-stream Transformer-based encoder to learn spatio-temporal action representations, which are then mapped to a semantic space to align with text embeddings. Multi-grained motion-text alignment incorporates contrastive learning at three levels: global instance alignment, stream-specific alignment, and fine-grained alignment. Extensive experiments on popular benchmarks with heterogeneous skeleton data demonstrate both the effectiveness and the generalization ability of the proposed method. Code is available at https://github.com/jidongkuang/Universal-Skeleton.

Related papers

• MotionDreamer: Universal Skeletal Motion Generation for 3D Rigged Shapes [52.45449797329548]
  MotionDreamer is a diffusion-based framework for category-agnostic skeletal animation generation from 2D video guidance.<n>We propose a structural-semantic injection mechanism to bridge the gap between 2D visual motion cues and heterogeneous 3D skeletal structures.<n>Our approach significantly outperforms existing methods, setting a new state-of-the-art benchmark for robust and efficient 4D asset generation.
  arXiv  Detail & Related papers  (2026-06-01T00:42:31Z)
• Generative Data Augmentation for Skeleton Action Recognition [12.151605559533337]
  We propose a conditional generative pipeline for data augmentation in skeleton action recognition.<n>Our method learns the distribution of real skeleton sequences under the constraint of action labels.<n>It can effectively generate skeleton sequences and achieve competitive recognition performance in low-data scenarios.
  arXiv  Detail & Related papers  (2026-04-16T12:20:29Z)
• Skeleton-to-Image Encoding: Enabling Skeleton Representation Learning via Vision-Pretrained Models [110.11712022072975]
  We introduce Skeleton-to-Image, a novel representation that transforms skeleton sequences into image-like data.<n>This encoding enables the use of powerful vision-pretrained models for self-supervised skeleton representation learning.
  arXiv  Detail & Related papers  (2026-03-06T06:54:55Z)
• Topology-Agnostic Animal Motion Generation from Text Prompt [16.557163253248817]
  We introduce OmniZoo, a large-scale animal motion dataset spanning 140 species and 32,979 sequences.<n>We propose a generalized autoregressive motion generation framework capable of producing text-driven motions for arbitrary skeletal topologies.
  arXiv  Detail & Related papers  (2025-12-11T07:08:29Z)
• Heterogeneous Skeleton-Based Action Representation Learning [12.690541950946358]
  This work addresses the challenge of heterogeneous skeleton-based action representation learning.<n>The proposed framework comprises two primary components: heterogeneous skeleton processing and unified representation learning.<n>Our approach can be applied to action recognition in robots with different humanoid structures.
  arXiv  Detail & Related papers  (2025-06-04T01:39:13Z)
• AnyTop: Character Animation Diffusion with Any Topology [54.07731933876742]
  We introduce AnyTop, a diffusion model that generates motions for diverse characters with distinct motion dynamics.<n>Our work features a transformer-based denoising network, tailored for arbitrary skeleton learning.<n>Our evaluation demonstrates that AnyTops well, even with as few as three training examples per topology, and can produce motions for unseen skeletons as well.
  arXiv  Detail & Related papers  (2025-02-24T17:00:36Z)
• Motif Guided Graph Transformer with Combinatorial Skeleton Prototype Learning for Skeleton-Based Person Re-Identification [60.939250172443586]
  Person re-identification (re-ID) via 3D skeleton data is a challenging task with significant value in many scenarios.<n>Existing skeleton-based methods typically assume virtual motion relations between all joints, and adopt average joint or sequence representations for learning.<n>This paper presents a generic Motif guided graph transformer with Combinatorial skeleton prototype learning (MoCos)<n>MoCos exploits structure-specific and gait-related body relations as well as features of skeleton graphs to learn effective skeleton representations for person re-ID.
  arXiv  Detail & Related papers  (2024-12-12T08:13:29Z)
• SkeleTR: Towrads Skeleton-based Action Recognition in the Wild [86.03082891242698]
  SkeleTR is a new framework for skeleton-based action recognition. It first models the intra-person skeleton dynamics for each skeleton sequence with graph convolutions. It then uses stacked Transformer encoders to capture person interactions that are important for action recognition in general scenarios.
  arXiv  Detail & Related papers  (2023-09-20T16:22:33Z)
• Skeleton-Aware Networks for Deep Motion Retargeting [83.65593033474384]
  We introduce a novel deep learning framework for data-driven motion between skeletons. Our approach learns how to retarget without requiring any explicit pairing between the motions in the training set.
  arXiv  Detail & Related papers  (2020-05-12T12:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.