StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition

• URL: http://arxiv.org/abs/2212.12857v2
• Date: Sun, 7 Apr 2024 06:34:37 GMT
• Title: StepNet: Spatial-temporal Part-aware Network for Isolated Sign Language Recognition
• Authors: Xiaolong Shen, Zhedong Zheng, Yi Yang,
• Abstract summary: We propose a new framework called Spatial-temporal Part-aware network(StepNet) based on RGB parts. Part-level Spatial Modeling automatically captures the appearance-based properties, such as hands and faces, in the feature space. Part-level Temporal Modeling implicitly mines the long-short term context to capture the relevant attributes over time.
• Score: 33.44126628779347
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The goal of sign language recognition (SLR) is to help those who are hard of hearing or deaf overcome the communication barrier. Most existing approaches can be typically divided into two lines, i.e., Skeleton-based and RGB-based methods, but both the two lines of methods have their limitations. Skeleton-based methods do not consider facial expressions, while RGB-based approaches usually ignore the fine-grained hand structure. To overcome both limitations, we propose a new framework called Spatial-temporal Part-aware network~(StepNet), based on RGB parts. As its name suggests, it is made up of two modules: Part-level Spatial Modeling and Part-level Temporal Modeling. Part-level Spatial Modeling, in particular, automatically captures the appearance-based properties, such as hands and faces, in the feature space without the use of any keypoint-level annotations. On the other hand, Part-level Temporal Modeling implicitly mines the long-short term context to capture the relevant attributes over time. Extensive experiments demonstrate that our StepNet, thanks to spatial-temporal modules, achieves competitive Top-1 Per-instance accuracy on three commonly-used SLR benchmarks, i.e., 56.89% on WLASL, 77.2% on NMFs-CSL, and 77.1% on BOBSL. Additionally, the proposed method is compatible with the optical flow input and can produce superior performance if fused. For those who are hard of hearing, we hope that our work can act as a preliminary step.

Related papers

• Bengali Sign Language Recognition through Hand Pose Estimation using Multi-Branch Spatial-Temporal Attention Model [0.5825410941577593]
  We propose a spatial-temporal attention-based BSL recognition model considering hand joint skeletons extracted from the sequence of images. Our model captures discriminative structural displacements and short-range dependency based on unified joint features projected onto high-dimensional feature space.
  arXiv  Detail & Related papers  (2024-08-26T08:55:16Z)
• Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion [57.232688209606515]
  We present HTCL, a novel Temporal Temporal Context Learning paradigm for improving camera-based semantic scene completion. Our method ranks $1st$ on the Semantic KITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU.
  arXiv  Detail & Related papers  (2024-07-02T09:11:17Z)
• Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition [10.048809585477555]
  Skeleton-aware sign language recognition has gained popularity due to its ability to remain unaffected by background information. Current methods utilize spatial graph modules and temporal modules to capture spatial and temporal features, respectively. We propose a new spatial architecture consisting of two concurrent branches, which build input-sensitive joint relationships. We then propose a new temporal module to model multi-scale temporal information to capture complex human dynamics.
  arXiv  Detail & Related papers  (2024-03-19T07:42:57Z)
• Implicit Temporal Modeling with Learnable Alignment for Video Recognition [95.82093301212964]
  We propose a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance. ILA achieves a top-1 accuracy of 88.7% on Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H.
  arXiv  Detail & Related papers  (2023-04-20T17:11:01Z)
• Part-aware Prototypical Graph Network for One-shot Skeleton-based Action Recognition [57.86960990337986]
  One-shot skeleton-based action recognition poses unique challenges in learning transferable representation from base classes to novel classes. We propose a part-aware prototypical representation for one-shot skeleton-based action recognition. We demonstrate the effectiveness of our method on two public skeleton-based action recognition datasets.
  arXiv  Detail & Related papers  (2022-08-19T04:54:56Z)
• Spatial Temporal Graph Attention Network for Skeleton-Based Action Recognition [10.60209288486904]
  It's common for current methods in skeleton-based action recognition to mainly consider capturing long-term temporal dependencies. We propose a general framework, coined as STGAT, to model cross-spacetime information flow. STGAT achieves state-of-the-art performance on three large-scale datasets.
  arXiv  Detail & Related papers  (2022-08-18T02:34:46Z)
• Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing [71.19528222206088]
  We propose a novel Decoupled Multi-task Learning with Cyclical Self-Regulation for face parsing. Specifically, DML-CSR designs a multi-task model which comprises face parsing, binary edge, and category edge detection. Our method achieves the new state-of-the-art performance on the Helen, CelebA-HQ, and LapaMask datasets.
  arXiv  Detail & Related papers  (2022-03-28T02:12:30Z)
• Denoised Non-Local Neural Network for Semantic Segmentation [18.84185406522064]
  We propose a Denoised Non-Local Network (Denoised NL) to eliminate the inter-class and intra-class noises respectively. Our proposed NL can achieve the state-of-the-art performance of 83.5% and 46.69% mIoU on Cityscapes and ADE20K, respectively.
  arXiv  Detail & Related papers  (2021-10-27T06:16:31Z)
• Sign Language Recognition via Skeleton-Aware Multi-Model Ensemble [71.97020373520922]
  Sign language is commonly used by deaf or mute people to communicate. We propose a novel Multi-modal Framework with a Global Ensemble Model (GEM) for isolated Sign Language Recognition ( SLR) Our proposed SAM- SLR-v2 framework is exceedingly effective and achieves state-of-the-art performance with significant margins.
  arXiv  Detail & Related papers  (2021-10-12T16:57:18Z)
• The Devil is in the Boundary: Exploiting Boundary Representation for Basis-based Instance Segmentation [85.153426159438]
  We propose Basis based Instance(B2Inst) to learn a global boundary representation that can complement existing global-mask-based methods. Our B2Inst leads to consistent improvements and accurately parses out the instance boundaries in a scene.
  arXiv  Detail & Related papers  (2020-11-26T11:26:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.