USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognition

• URL: http://arxiv.org/abs/2512.13415v1
• Date: Mon, 15 Dec 2025 15:05:16 GMT
• Title: USTM: Unified Spatial and Temporal Modeling for Continuous Sign Language Recognition
• Authors: Ahmed Abul Hasanaath, Hamzah Luqman,
• Abstract summary: Continuous sign language recognition requires precise modeling-temporal to accurately recognize sequences of gestures in videos.<n>These techniques fail in capturing fine-grained hand and facial cues and modeling long-range temporal dependencies.<n>We propose the Unified Unified S-temporal Modeling (USTM) framework to address these limitations.<n>Our framework captures fine-grained spatial features alongside short and long-term temporal context, enabling robust sign language recognition from RGB videos without relying on multi-stream inputs or auxiliary modalities.
• Score: 3.8100688074986095
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous sign language recognition (CSLR) requires precise spatio-temporal modeling to accurately recognize sequences of gestures in videos. Existing frameworks often rely on CNN-based spatial backbones combined with temporal convolution or recurrent modules. These techniques fail in capturing fine-grained hand and facial cues and modeling long-range temporal dependencies. To address these limitations, we propose the Unified Spatio-Temporal Modeling (USTM) framework, a spatio-temporal encoder that effectively models complex patterns using a combination of a Swin Transformer backbone enhanced with lightweight temporal adapter with positional embeddings (TAPE). Our framework captures fine-grained spatial features alongside short and long-term temporal context, enabling robust sign language recognition from RGB videos without relying on multi-stream inputs or auxiliary modalities. Extensive experiments on benchmarked datasets including PHOENIX14, PHOENIX14T, and CSL-Daily demonstrate that USTM achieves state-of-the-art performance against RGB-based as well as multi-modal CSLR approaches, while maintaining competitive performance against multi-stream approaches. These results highlight the strength and efficacy of the USTM framework for CSLR. The code is available at https://github.com/gufranSabri/USTM

Related papers

• TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition [59.99922360648663]
  TSkel-Mamba is a hybrid Transformer-Mamba framework that effectively captures both spatial and temporal dynamics.<n>The MTI module employs multi-scale Cycle operators to capture cross-channel temporal interactions, a critical factor in action recognition.
  arXiv  Detail & Related papers  (2025-12-12T11:55:16Z)
• Uniform Discrete Diffusion with Metric Path for Video Generation [103.86033350602908]
  Continuous-space video generation has advanced rapidly, while discrete approaches lag behind due to error accumulation and long-duration inconsistency.<n>We present Uniform generative modeling and present Uniform pAth (URSA), a powerful framework that bridges the gap with continuous approaches for scalable video generation.<n>URSA consistently outperforms existing discrete methods and achieves performance comparable to state-of-the-art continuous diffusion methods.
  arXiv  Detail & Related papers  (2025-10-28T17:59:57Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• Trajectory-aware Shifted State Space Models for Online Video Super-Resolution [57.87099307245989]
  This paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba)<n>TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames.<n>Our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% reduction complexity (in MACs)
  arXiv  Detail & Related papers  (2025-08-14T08:42:15Z)
• Adapting LLMs to Time Series Forecasting via Temporal Heterogeneity Modeling and Semantic Alignment [32.41581846555808]
  Large Language Models (LLMs) have recently demonstrated impressive capabilities in natural language processing.<n>We propose TALON, a unified framework that enhances LLM-based forecasting by modeling temporal and enforcing semantic alignment.<n>Experiments on seven real-world benchmarks demonstrate that TALON achieves superior performance across all datasets.
  arXiv  Detail & Related papers  (2025-08-10T06:06:19Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• Dynamic Spatial-Temporal Aggregation for Skeleton-Aware Sign Language Recognition [10.048809585477555]
  Skeleton-aware sign language recognition has gained popularity due to its ability to remain unaffected by background information. Current methods utilize spatial graph modules and temporal modules to capture spatial and temporal features, respectively. We propose a new spatial architecture consisting of two concurrent branches, which build input-sensitive joint relationships. We then propose a new temporal module to model multi-scale temporal information to capture complex human dynamics.
  arXiv  Detail & Related papers  (2024-03-19T07:42:57Z)
• Implicit Temporal Modeling with Learnable Alignment for Video Recognition [95.82093301212964]
  We propose a novel Implicit Learnable Alignment (ILA) method, which minimizes the temporal modeling effort while achieving incredibly high performance. ILA achieves a top-1 accuracy of 88.7% on Kinetics-400 with much fewer FLOPs compared with Swin-L and ViViT-H.
  arXiv  Detail & Related papers  (2023-04-20T17:11:01Z)
• Multi-scale temporal network for continuous sign language recognition [10.920363368754721]
  Continuous Sign Language Recognition is a challenging research task due to the lack of accurate annotation on the temporal sequence of sign language data. This paper proposes a multi-scale temporal network (MSTNet) to extract more accurate temporal features. Experimental results on two publicly available datasets demonstrate that our method can effectively extract sign language features in an end-to-end manner without any prior knowledge.
  arXiv  Detail & Related papers  (2022-04-08T06:14:22Z)
• Multi-Scale Local-Temporal Similarity Fusion for Continuous Sign Language Recognition [4.059599144668737]
  Continuous sign language recognition is a public significant task that transcribes a sign language video into an ordered gloss sequence. One promising way is to adopt a one-dimensional convolutional network (1D-CNN) to temporally fuse the sequential frames. We propose to adaptively fuse local features via temporal similarity for this task.
  arXiv  Detail & Related papers  (2021-07-27T12:06:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.