GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition

• URL: http://arxiv.org/abs/2407.14812v1
• Date: Sat, 20 Jul 2024 09:05:17 GMT
• Title: GaitMA: Pose-guided Multi-modal Feature Fusion for Gait Recognition
• Authors: Fanxu Min, Shaoxiang Guo, Fan Hao, Junyu Dong,
• Abstract summary: Gait recognition is a biometric technology that recognizes the identity of humans through their walking patterns. We propose a novel gait recognition framework, dubbed Gait Multi-model Aggregation Network (GaitMA) First, skeletons are represented by joint/limb-based heatmaps, and features from silhouettes and skeletons are respectively extracted using two CNN-based feature extractors.
• Score: 26.721242606715354
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Gait recognition is a biometric technology that recognizes the identity of humans through their walking patterns. Existing appearance-based methods utilize CNN or Transformer to extract spatial and temporal features from silhouettes, while model-based methods employ GCN to focus on the special topological structure of skeleton points. However, the quality of silhouettes is limited by complex occlusions, and skeletons lack dense semantic features of the human body. To tackle these problems, we propose a novel gait recognition framework, dubbed Gait Multi-model Aggregation Network (GaitMA), which effectively combines two modalities to obtain a more robust and comprehensive gait representation for recognition. First, skeletons are represented by joint/limb-based heatmaps, and features from silhouettes and skeletons are respectively extracted using two CNN-based feature extractors. Second, a co-attention alignment module is proposed to align the features by element-wise attention. Finally, we propose a mutual learning module, which achieves feature fusion through cross-attention, Wasserstein loss is further introduced to ensure the effective fusion of two modalities. Extensive experimental results demonstrate the superiority of our model on Gait3D, OU-MVLP, and CASIA-B.

Related papers

• GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning [51.677086019209554]
  We propose a Generalized Structural Sparse to capture powerful relationships across modalities for pair-wise similarity learning. The distance metric delicately encapsulates two formats of diagonal and block-diagonal terms. Experiments on cross-modal and two extra uni-modal retrieval tasks have validated its superiority and flexibility.
  arXiv  Detail & Related papers  (2024-10-20T03:45:50Z)
• ZipGait: Bridging Skeleton and Silhouette with Diffusion Model for Advancing Gait Recognition [31.732554267037305]
  We make the first attempt to reconstruct dense body shapes from discrete skeleton distributions via the diffusion model. We introduce Perception Gait Integration (PGI) to integrate different gait features through a two-stage process. ZipGait demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both cross-domain and intra-domain settings.
  arXiv  Detail & Related papers  (2024-08-22T03:52:44Z)
• One-Shot Action Recognition via Multi-Scale Spatial-Temporal Skeleton Matching [77.6989219290789]
  One-shot skeleton action recognition aims to learn a skeleton action recognition model with a single training sample. This paper presents a novel one-shot skeleton action recognition technique that handles skeleton action recognition via multi-scale spatial-temporal feature matching.
  arXiv  Detail & Related papers  (2023-07-14T11:52:10Z)
• Part-aware Prototypical Graph Network for One-shot Skeleton-based Action Recognition [57.86960990337986]
  One-shot skeleton-based action recognition poses unique challenges in learning transferable representation from base classes to novel classes. We propose a part-aware prototypical representation for one-shot skeleton-based action recognition. We demonstrate the effectiveness of our method on two public skeleton-based action recognition datasets.
  arXiv  Detail & Related papers  (2022-08-19T04:54:56Z)
• Towards a Deeper Understanding of Skeleton-based Gait Recognition [4.812321790984493]
  In recent years, most gait recognition methods used the person's silhouette to extract the gait features. Model-based methods do not suffer from these problems and are able to represent the temporal motion of body joints. In this work, we propose an approach based on Graph Convolutional Networks (GCNs) that combines higher-order inputs, and residual networks.
  arXiv  Detail & Related papers  (2022-04-16T18:23:37Z)
• Combining the Silhouette and Skeleton Data for Gait Recognition [13.345465199699]
  Two dominant gait recognition works are appearance-based and model-based, which extract features from silhouettes and skeletons, respectively. This paper proposes a CNN-based branch taking silhouettes as input and a GCN-based branch taking skeletons as input. For better gait representation in the GCN-based branch, we present a fully connected graph convolution operator to integrate multi-scale graph convolutions.
  arXiv  Detail & Related papers  (2022-02-22T03:21:51Z)
• Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition [140.18376685167857]
  A simple yet effective multi-scale semantics-guided neural network is proposed for skeleton-based action recognition. MS-SGN achieves the state-of-the-art performance on the NTU60, NTU120, and SYSU datasets.
  arXiv  Detail & Related papers  (2021-11-07T03:50:50Z)
• TNTC: two-stream network with transformer-based complementarity for gait-based emotion recognition [4.9752798133038585]
  gait-based emotion recognition, especially gait skeletons-based characteristic, has attracted much attention. We propose a novel two-stream network with transformer-based complementarity, termed as TNTC. A new transformer-based complementarity module (TCM) is proposed to bridge the complementarity between two streams hierarchically.
  arXiv  Detail & Related papers  (2021-10-26T13:55:31Z)
• Learning Rich Features for Gait Recognition by Integrating Skeletons and Silhouettes [20.766540020533803]
  This paper proposes a simple yet effective bimodal fusion network, which mines the complementary clues of skeletons and silhouettes to learn rich features for gait identification. Under the most challenging condition of walking in different clothes on CASIA-B, our method achieves the rank-1 accuracy of 92.1%.
  arXiv  Detail & Related papers  (2021-10-26T04:42:24Z)
• Learning Multi-Granular Spatio-Temporal Graph Network for Skeleton-based Action Recognition [49.163326827954656]
  We propose a novel multi-granular-temporal graph network for skeleton-based action classification. We develop a dual-head graph network consisting of two inter-leaved branches, which enables us to extract at least two-temporal resolutions. We conduct extensive experiments on three large-scale datasets.
  arXiv  Detail & Related papers  (2021-08-10T09:25:07Z)
• What and Where: Modeling Skeletons from Semantic and Spatial Perspectives for Action Recognition [46.836815779215456]
  We propose to model skeletons from a novel spatial perspective, from which the model takes the spatial location as prior knowledge to group human joints. From the semantic perspective, we propose a Transformer-like network that is expert in modeling joint correlations. From the spatial perspective, we transform the skeleton data into the sparse format for efficient feature extraction.
  arXiv  Detail & Related papers  (2020-04-07T10:53:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.