A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics

• URL: http://arxiv.org/abs/2604.15215v1
• Date: Thu, 16 Apr 2026 16:47:08 GMT
• Title: A Hierarchical Spatiotemporal Action Tokenizer for In-Context Imitation Learning in Robotics
• Abstract summary: We present a novel hierarchical action tokenizer for incontext- imitation learning, namely HiST-AT. Specifically, our hierarchical-temporal approach conducts multi-level clustering, while simultaneously recovering input actions and their associated timestamps cues.
• Score: 6.4420182172671865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel hierarchical spatiotemporal action tokenizer for in-context imitation learning. We first propose a hierarchical approach, which consists of two successive levels of vector quantization. In particular, the lower level assigns input actions to fine-grained subclusters, while the higher level further maps fine-grained subclusters to clusters. Our hierarchical approach outperforms the non-hierarchical counterpart, while mainly exploiting spatial information by reconstructing input actions. Furthermore, we extend our approach by utilizing both spatial and temporal cues, forming a hierarchical spatiotemporal action tokenizer, namely HiST-AT. Specifically, our hierarchical spatiotemporal approach conducts multi-level clustering, while simultaneously recovering input actions and their associated timestamps. Finally, extensive evaluations on multiple simulation and real robotic manipulation benchmarks show that our approach establishes a new state-of-the-art performance in in-context imitation learning.

Related papers

• Online Continual Learning with Dynamic Label Hierarchies [72.33335166136045]
  We introduce a new problem setting, DHOCL, where severity evolve across granularities and each sample provides supervision at a single hierarchical level.<n>In this setting, we find two fundamental issues: (i) partial supervision under mixed granularities provides only point-wise signals over an evolving path-wise hierarchy, which constrains plasticity and undermines cross-level semantic consistency.<n>We propose HALO, which adaptively combines complementary classification heads, regularized by organized learnable hierarchical prototypes, enabling rapid adaptation, hierarchical consistency, and structured knowledge consolidation as the taxonomy evolves.
  arXiv  Detail & Related papers  (2026-05-12T08:20:23Z)
• Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization [4.446876533427628]
  We propose a novel hierarchical temporal vectorization framework for unsupervised skeleton-based temporal action segmentation.<n>We demonstrate that our approach establishes a new state-of-the-art performance and reduces segment length bias in unsupervised skeleton-based temporal action segmentation.
  arXiv  Detail & Related papers  (2026-04-16T16:24:40Z)
• Feature Identification for Hierarchical Contrastive Learning [7.655211354400059]
  We propose two novel hierarchical contrastive learning (HMLC) methods.<n>Our approach explicitly models inter-class relationships and imbalanced class distribution at higher hierarchy levels.<n>Our method achieves state-of-the-art performance in linear evaluation, outperforming existing hierarchical contrastive learning methods by 2 percentage points in terms of accuracy.
  arXiv  Detail & Related papers  (2025-10-01T12:46:47Z)
• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation.<n>Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy.<n>Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• Temporal-aware Hierarchical Mask Classification for Video Semantic Segmentation [62.275143240798236]
  Video semantic segmentation dataset has limited categories per video. Less than 10% of queries could be matched to receive meaningful gradient updates during VSS training. Our method achieves state-of-the-art performance on the latest challenging VSS benchmark VSPW without bells and whistles.
  arXiv  Detail & Related papers  (2023-09-14T20:31:06Z)
• Hierarchical Spatio-Temporal Representation Learning for Gait Recognition [6.877671230651998]
  Gait recognition is a biometric technique that identifies individuals by their unique walking styles. We propose a hierarchical-temporal representation learning framework for extracting gait features from coarse to fine. Our method outperforms the state-of-the-art while maintaining a reasonable balance between model accuracy and complexity.
  arXiv  Detail & Related papers  (2023-07-19T09:30:00Z)
• CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learning [25.84621883831624]
  CRISP is a curriculum-driven framework that tackles instability in hierarchical reinforcement learning.<n>It adaptively re-labels expert demonstrations to always generate reachable subgoals by the current low-level primitive.<n>It improves success rates by more than 40% over strong hierarchical and flat baselines.
  arXiv  Detail & Related papers  (2023-04-07T08:22:50Z)
• Spatio-temporal Relation Modeling for Few-shot Action Recognition [100.3999454780478]
  We propose a few-shot action recognition framework, STRM, which enhances class-specific featureriminability while simultaneously learning higher-order temporal representations. Our approach achieves an absolute gain of 3.5% in classification accuracy, as compared to the best existing method in the literature.
  arXiv  Detail & Related papers  (2021-12-09T18:59:14Z)
• Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos [6.187780920448871]
  This paper focuses on task recognition and action segmentation in weakly-labeled instructional videos. We propose a two-stream framework, which exploits semantic and temporal hierarchies to recognize top-level tasks in instructional videos. We present a novel top-down weakly-supervised action segmentation approach, where the predicted task is used to constrain the inference of fine-grained action sequences.
  arXiv  Detail & Related papers  (2021-10-12T02:32:15Z)
• Modeling long-term interactions to enhance action recognition [81.09859029964323]
  We propose a new approach to under-stand actions in egocentric videos that exploits the semantics of object interactions at both frame and temporal levels. We use a region-based approach that takes as input a primary region roughly corresponding to the user hands and a set of secondary regions potentially corresponding to the interacting objects. The proposed approach outperforms the state-of-the-art in terms of action recognition on standard benchmarks.
  arXiv  Detail & Related papers  (2021-04-23T10:08:15Z)
• Hierarchically Decoupled Spatial-Temporal Contrast for Self-supervised Video Representation Learning [6.523119805288132]
  We present a novel technique for self-supervised video representation learning by: (a) decoupling the learning objective into two contrastive subtasks respectively emphasizing spatial and temporal features, and (b) performing it hierarchically to encourage multi-scale understanding.
  arXiv  Detail & Related papers  (2020-11-23T08:05:39Z)
• MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation [87.16030562892537]
  We propose a multi-stage architecture for the temporal action segmentation task. The first stage generates an initial prediction that is refined by the next ones. Our models achieve state-of-the-art results on three datasets.
  arXiv  Detail & Related papers  (2020-06-16T14:50:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.