Accurate online action and gesture recognition system using detectors and Deep SPD Siamese Networks

• URL: http://arxiv.org/abs/2511.05250v1
• Date: Fri, 07 Nov 2025 14:09:43 GMT
• Title: Accurate online action and gesture recognition system using detectors and Deep SPD Siamese Networks
• Abstract summary: We propose an online recognition system for skeleton sequence streaming composed from two main components: a detector and a classifier.<n>The detector is flexible and able to identify the kinetic state continuously.<n>We conduct extensive experiments on both hand gesture and body action recognition benchmarks to prove the accuracy of our online recognition system.
• Score: 4.667044415202222
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Online continuous motion recognition is a hot topic of research since it is more practical in real life application cases. Recently, Skeleton-based approaches have become increasingly popular, demonstrating the power of using such 3D temporal data. However, most of these works have focused on segment-based recognition and are not suitable for the online scenarios. In this paper, we propose an online recognition system for skeleton sequence streaming composed from two main components: a detector and a classifier, which use a Semi-Positive Definite (SPD) matrix representation and a Siamese network. The powerful statistical representations for the skeletal data given by the SPD matrices and the learning of their semantic similarity by the Siamese network enable the detector to predict time intervals of the motions throughout an unsegmented sequence. In addition, they ensure the classifier capability to recognize the motion in each predicted interval. The proposed detector is flexible and able to identify the kinetic state continuously. We conduct extensive experiments on both hand gesture and body action recognition benchmarks to prove the accuracy of our online recognition system which in most cases outperforms state-of-the-art performances.

Related papers

• HOI-aware Adaptive Network for Weakly-supervised Action Segmentation [64.63922024617493]
  We propose an HOI-aware adaptive network named AdaAct for weakly-supervised action segmentation.<n>We exploit temporally global but spatially local human-object interactions (HOI) as video-level prior knowledge for action segmentation.
  arXiv  Detail & Related papers  (2026-04-29T02:11:51Z)
• Dynamic LIBRAS Gesture Recognition via CNN over Spatiotemporal Matrix Representation [0.0]
  This paper proposes a method for dynamic hand gesture recognition based on the composition of two models: the MediaPipe Hander, and a convolutional neural network (CNN)<n>The method is applied to recognition of RASLIB (Brazilian Sign Language) gestures for device control in a home automation system, covering 11 classes of static and dynamic gestures.<n>Tests achieved 95% accuracy under low-light conditions and 92% under normal lighting.
  arXiv  Detail & Related papers  (2026-03-26T19:37:28Z)
• Informative Sample Selection Model for Skeleton-based Action Recognition with Limited Training Samples [51.59753385094941]
  3D Action Recognition with Limited Training Samples, also known as semi-supervised 3D Action Recognition, has been proposed.<n>We reformulate Semi-supervised 3D action recognition via active learning from a novel perspective by casting it as a Markov Decision Process (MDP)<n>To enhance the representational capacity of the factors in the state-action pairs within our method, we project them from Euclidean space to hyperbolic space.
  arXiv  Detail & Related papers  (2025-10-29T10:03:33Z)
• Online hand gesture recognition using Continual Graph Transformers [1.3927943269211591]
  We propose a novel online recognition system designed for real-time skeleton sequence streaming.<n>Our approach achieves state-of-the-art accuracy and significantly reduces false positive rates, making it a compelling solution for real-time applications.<n>The proposed system can be seamlessly integrated into various domains, including human-robot collaboration and assistive technologies.
  arXiv  Detail & Related papers  (2025-02-20T17:27:55Z)
• Joint-bone Fusion Graph Convolutional Network for Semi-supervised Skeleton Action Recognition [65.78703941973183]
  We propose a novel correlation-driven joint-bone fusion graph convolutional network (CD-JBF-GCN) as an encoder and use a pose prediction head as a decoder. Specifically, the CD-JBF-GC can explore the motion transmission between the joint stream and the bone stream. The pose prediction based auto-encoder in the self-supervised training stage allows the network to learn motion representation from unlabeled data.
  arXiv  Detail & Related papers  (2022-02-08T16:03:15Z)
• Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems [0.0]
  Long-term motion patterns alone play a pivotal role in the task of recognizing an event. We show that the long-term motion patterns alone play a pivotal role in the task of recognizing an event. Only the temporal features are exploited using a hybrid Convolutional Neural Network (CNN) + Recurrent Neural Network (RNN) architecture.
  arXiv  Detail & Related papers  (2021-11-03T08:30:38Z)
• Efficient Modelling Across Time of Human Actions and Interactions [92.39082696657874]
  We argue that current fixed-sized-temporal kernels in 3 convolutional neural networks (CNNDs) can be improved to better deal with temporal variations in the input. We study how we can better handle between classes of actions, by enhancing their feature differences over different layers of the architecture. The proposed approaches are evaluated on several benchmark action recognition datasets and show competitive results.
  arXiv  Detail & Related papers  (2021-10-05T15:39:11Z)
• Sequential convolutional network for behavioral pattern extraction in gait recognition [0.7874708385247353]
  We propose a sequential convolutional network (SCN) to learn the walking pattern of individuals. In SCN, behavioral information extractors (BIE) are constructed to comprehend intermediate feature maps in time series. A multi-frame aggregator in SCN performs feature integration on a sequence whose length is uncertain, via a mobile 3D convolutional layer.
  arXiv  Detail & Related papers  (2021-04-23T08:44:10Z)
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery [84.73764603474413]
  We propose a novel online approach of multi-modal graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset.
  arXiv  Detail & Related papers  (2020-11-03T11:00:10Z)
• Memory Group Sampling Based Online Action Recognition Using Kinetic Skeleton Features [4.674689979981502]
  We propose two core ideas to handle the online action recognition problem. First, we combine the spatial and temporal skeleton features to depict the actions. Second, we propose a memory group sampling method to combine the previous action frames and current action frames. Third, an improved 1D CNN network is employed for training and testing using the features from sampled frames.
  arXiv  Detail & Related papers  (2020-11-01T16:43:08Z)
• A Prospective Study on Sequence-Driven Temporal Sampling and Ego-Motion Compensation for Action Recognition in the EPIC-Kitchens Dataset [68.8204255655161]
  Action recognition is one of the top-challenging research fields in computer vision. ego-motion recorded sequences have become of important relevance. The proposed method aims to cope with it by estimating this ego-motion or camera motion.
  arXiv  Detail & Related papers  (2020-08-26T14:44:45Z)
• Complex Human Action Recognition in Live Videos Using Hybrid FR-DL Method [1.027974860479791]
  We address challenges of the preprocessing phase, by an automated selection of representative frames among the input sequences. We propose a hybrid technique using background subtraction and HOG, followed by application of a deep neural network and skeletal modelling method. We name our model as Feature Reduction & Deep Learning based action recognition method, or FR-DL in short.
  arXiv  Detail & Related papers  (2020-07-06T15:12:50Z)
• Temporal Pyramid Network for Action Recognition [129.12076009042622]
  We propose a generic Temporal Pyramid Network (TPN) at the feature-level, which can be flexibly integrated into 2D or 3D backbone networks. TPN shows consistent improvements over other challenging baselines on several action recognition datasets.
  arXiv  Detail & Related papers  (2020-04-07T17:17:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.