SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval

• URL: http://arxiv.org/abs/2407.16394v1
• Date: Tue, 23 Jul 2024 11:31:11 GMT
• Title: SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval
• Authors: Longtao Jiang, Min Wang, Zecheng Li, Yao Fang, Wengang Zhou, Houqiang Li,
• Abstract summary: Previous works typically only encode RGB videos to obtain high-level semantic features. Existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training. We propose a novel sign language representation framework called Semantically Enhanced Dual-Stream.
• Score: 82.51117533271517
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Different from traditional video retrieval, sign language retrieval is more biased towards understanding the semantic information of human actions contained in video clips. Previous works typically only encode RGB videos to obtain high-level semantic features, resulting in local action details drowned in a large amount of visual information redundancy. Furthermore, existing RGB-based sign retrieval works suffer from the huge memory cost of dense visual data embedding in end-to-end training, and adopt offline RGB encoder instead, leading to suboptimal feature representation. To address these issues, we propose a novel sign language representation framework called Semantically Enhanced Dual-Stream Encoder (SEDS), which integrates Pose and RGB modalities to represent the local and global information of sign language videos. Specifically, the Pose encoder embeds the coordinates of keypoints corresponding to human joints, effectively capturing detailed action features. For better context-aware fusion of two video modalities, we propose a Cross Gloss Attention Fusion (CGAF) module to aggregate the adjacent clip features with similar semantic information from intra-modality and inter-modality. Moreover, a Pose-RGB Fine-grained Matching Objective is developed to enhance the aggregated fusion feature by contextual matching of fine-grained dual-stream features. Besides the offline RGB encoder, the whole framework only contains learnable lightweight networks, which can be trained end-to-end. Extensive experiments demonstrate that our framework significantly outperforms state-of-the-art methods on various datasets.

Related papers

• ViDSOD-100: A New Dataset and a Baseline Model for RGB-D Video Salient Object Detection [51.16181295385818]
  We first collect an annotated RGB-D video SODOD (DSOD-100) dataset, which contains 100 videos within a total of 9,362 frames. All the frames in each video are manually annotated to a high-quality saliency annotation. We propose a new baseline model, named attentive triple-fusion network (ATF-Net) for RGB-D salient object detection.
  arXiv  Detail & Related papers  (2024-06-18T12:09:43Z)
• Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models [15.231177830711077]
  We introduce a novel pattern recognition framework that consolidates semantic labels, RGB frames, and event streams. To handle the semantic labels, we convert them into language descriptions through prompt engineering. We integrate the RGB/Event features and semantic features using multimodal Transformer networks.
  arXiv  Detail & Related papers  (2023-11-30T14:35:51Z)
• Two-Stream Network for Sign Language Recognition and Translation [38.43767031555092]
  We introduce a dual visual encoder containing two separate streams to model both the raw videos and the keypoint sequences. The resulting model is called TwoStream- SLR, which is competent for sign language recognition. TwoStream-SLT is extended to a sign language translation model, TwoStream-SLT, by simply attaching an extra translation network.
  arXiv  Detail & Related papers  (2022-11-02T17:59:58Z)
• Robust Double-Encoder Network for RGB-D Panoptic Segmentation [31.807572107839576]
  Panoptic segmentation provides an interpretation of the scene by computing a pixelwise semantic label together with instance IDs. We propose a novel encoder-decoder neural network that processes RGB and depth separately through two encoders. We show that our approach achieves superior results compared to other common approaches for panoptic segmentation.
  arXiv  Detail & Related papers  (2022-10-06T11:46:37Z)
• Attention-based Dual Supervised Decoder for RGBD Semantic Segmentation [16.721758280029302]
  We propose a novel attention-based dual supervised decoder for RGBD semantic segmentation. In the encoder, we design a simple yet effective attention-based multimodal fusion module to extract and fuse deeply multi-level paired complementary information. Our method achieves superior performance against the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-01-05T03:12:27Z)
• Specificity-preserving RGB-D Saliency Detection [103.3722116992476]
  We propose a specificity-preserving network (SP-Net) for RGB-D saliency detection. Two modality-specific networks and a shared learning network are adopted to generate individual and shared saliency maps. Experiments on six benchmark datasets demonstrate that our SP-Net outperforms other state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-18T14:14:22Z)
• Cross-modality Discrepant Interaction Network for RGB-D Salient Object Detection [78.47767202232298]
  We propose a novel Cross-modality Discrepant Interaction Network (CDINet) for RGB-D SOD. Two components are designed to implement the effective cross-modality interaction. Our network outperforms $15$ state-of-the-art methods both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-04T11:24:42Z)
• Adaptive Intermediate Representations for Video Understanding [50.64187463941215]
  We introduce a new way to leverage semantic segmentation as an intermediate representation for video understanding. We propose a general framework which learns the intermediate representations (optical flow and semantic segmentation) jointly with the final video understanding task. We obtain more powerful visual representations for videos which lead to performance gains over the state-of-the-art.
  arXiv  Detail & Related papers  (2021-04-14T21:37:23Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.