Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization

• URL: http://arxiv.org/abs/2604.15196v1
• Date: Thu, 16 Apr 2026 16:24:40 GMT
• Title: Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization
• Abstract summary: We propose a novel hierarchical temporal vectorization framework for unsupervised skeleton-based temporal action segmentation.<n>We demonstrate that our approach establishes a new state-of-the-art performance and reduces segment length bias in unsupervised skeleton-based temporal action segmentation.
• Score: 4.446876533427628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a novel hierarchical spatiotemporal vector quantization framework for unsupervised skeleton-based temporal action segmentation. We first introduce a hierarchical approach, which includes two consecutive levels of vector quantization. Specifically, the lower level associates skeletons with fine-grained subactions, while the higher level further aggregates subactions into action-level representations. Our hierarchical approach outperforms the non-hierarchical baseline, while primarily exploiting spatial cues by reconstructing input skeletons. Next, we extend our approach by leveraging both spatial and temporal information, yielding a hierarchical spatiotemporal vector quantization scheme. In particular, our hierarchical spatiotemporal approach performs multi-level clustering, while simultaneously recovering input skeletons and their corresponding timestamps. Lastly, extensive experiments on multiple benchmarks, including HuGaDB, LARa, and BABEL, demonstrate that our approach establishes a new state-of-the-art performance and reduces segment length bias in unsupervised skeleton-based temporal action segmentation.

Related papers

• A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics [6.4420182172671865]
  We present a novel hierarchical action tokenizer for incontext- imitation learning, namely HiST-AT. Specifically, our hierarchical-temporal approach conducts multi-level clustering, while simultaneously recovering input actions and their associated timestamps cues.
  arXiv  Detail & Related papers  (2026-04-16T16:47:08Z)
• Skeleton Motion Words for Unsupervised Skeleton-Based Temporal Action Segmentation [11.045126693185377]
  We propose a novel approach for unsupervised skeleton-based temporal action segmentation.<n>Our method utilizes a sequence-to-sequence temporal autoencoder that keeps the information of the different joints disentangled in the embedding space.<n>We thoroughly evaluate the proposed approach on three widely used skeleton-based datasets.
  arXiv  Detail & Related papers  (2025-08-06T14:56:38Z)
• Hierarchical Vector Quantization for Unsupervised Action Segmentation [13.76466884074359]
  We address unsupervised temporal action segmentation, which segments a set of long ASM videos into semantically meaningful segments that are consistent across videos.<n>We propose a novel method, termed Hierarchical Vector Quantization (HVQ), that consists of two subsequent vector quantization modules.<n>Our approach outperforms the state of the art in terms of F1 score, recall and JSD.
  arXiv  Detail & Related papers  (2024-12-23T15:18:24Z)
• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation.<n>Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy.<n>Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy [67.45518210171024]
  Dynamic computation methods have shown notable acceleration for Large Language Models (LLMs) by skipping several layers of computations. We propose a Unified Layer Skipping strategy, which selects the number of layers to skip computation based solely on the target speedup ratio. Experimental results on two common tasks, i.e., machine translation and text summarization, indicate that given a target speedup ratio, the Unified Layer Skipping strategy significantly enhances both the inference performance and the actual model throughput.
  arXiv  Detail & Related papers  (2024-04-10T12:12:07Z)
• Skeleton2vec: A Self-supervised Learning Framework with Contextualized Target Representations for Skeleton Sequence [56.092059713922744]
  We show that using high-level contextualized features as prediction targets can achieve superior performance. Specifically, we propose Skeleton2vec, a simple and efficient self-supervised 3D action representation learning framework. Our proposed Skeleton2vec outperforms previous methods and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2024-01-01T12:08:35Z)
• Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent [46.86939432189035]
  We propose Anchor-based Multi-view Subspace Clustering with Hierarchical Feature Descent. Our proposed model consistently outperforms the state-of-the-art techniques.
  arXiv  Detail & Related papers  (2023-10-11T03:29:13Z)
• Hierarchical Spatio-Temporal Representation Learning for Gait Recognition [6.877671230651998]
  Gait recognition is a biometric technique that identifies individuals by their unique walking styles. We propose a hierarchical-temporal representation learning framework for extracting gait features from coarse to fine. Our method outperforms the state-of-the-art while maintaining a reasonable balance between model accuracy and complexity.
  arXiv  Detail & Related papers  (2023-07-19T09:30:00Z)
• Hierarchically Self-Supervised Transformer for Human Skeleton Representation Learning [45.13060970066485]
  We propose a self-supervised hierarchical pre-training scheme incorporated into a hierarchical Transformer-based skeleton sequence encoder (Hi-TRS) Under both supervised and semi-supervised evaluation protocols, our method achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-07-20T04:21:05Z)
• Modeling long-term interactions to enhance action recognition [81.09859029964323]
  We propose a new approach to under-stand actions in egocentric videos that exploits the semantics of object interactions at both frame and temporal levels. We use a region-based approach that takes as input a primary region roughly corresponding to the user hands and a set of secondary regions potentially corresponding to the interacting objects. The proposed approach outperforms the state-of-the-art in terms of action recognition on standard benchmarks.
  arXiv  Detail & Related papers  (2021-04-23T10:08:15Z)
• MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation [87.16030562892537]
  We propose a multi-stage architecture for the temporal action segmentation task. The first stage generates an initial prediction that is refined by the next ones. Our models achieve state-of-the-art results on three datasets.
  arXiv  Detail & Related papers  (2020-06-16T14:50:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.