Emotion Recognition in Signers

• URL: http://arxiv.org/abs/2512.15376v1
• Date: Wed, 17 Dec 2025 12:26:21 GMT
• Title: Emotion Recognition in Signers
• Abstract summary: We use eJSL, a new benchmark dataset for emotion recognition in Japanese Sign Language signers, and BOBSL, a large British Sign Language dataset with subtitles.<n>In eJSL, two signers expressed 78 distinct utterances with each of seven different emotional states, resulting in 1,092 video clips.<n>We empirically demonstrate that 1) textual emotion recognition in spoken language mitigates data scarcity in sign language, 2) temporal segment selection has a significant impact, and 3) incorporating hand motion enhances emotion recognition in signers.
• Score: 5.040463208115642
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recognition of signers' emotions suffers from one theoretical challenge and one practical challenge, namely, the overlap between grammatical and affective facial expressions and the scarcity of data for model training. This paper addresses these two challenges in a cross-lingual setting using our eJSL dataset, a new benchmark dataset for emotion recognition in Japanese Sign Language signers, and BOBSL, a large British Sign Language dataset with subtitles. In eJSL, two signers expressed 78 distinct utterances with each of seven different emotional states, resulting in 1,092 video clips. We empirically demonstrate that 1) textual emotion recognition in spoken language mitigates data scarcity in sign language, 2) temporal segment selection has a significant impact, and 3) incorporating hand motion enhances emotion recognition in signers. Finally we establish a stronger baseline than spoken language LLMs.

Related papers

• Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment [84.39962912136525]
  We develop a model for sign language understanding that performs sign language translation (SLT) and sign-subtitle alignment (SSA)<n>Our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA.
  arXiv  Detail & Related papers  (2025-12-08T21:05:46Z)
• EASL: Multi-Emotion Guided Semantic Disentanglement for Expressive Sign Language Generation [7.76229483761977]
  We propose EASL (Emotion-Aware Sign Language), a multi-emotion-guided generation architecture for fine-grained emotional integration.<n>We introduce emotion-semantic disentanglement modules with progressive training to separately extract semantic and affective features.<n>During pose decoding, the emotional representations guide semantic interaction to generate sign poses with 7-class emotion confidence scores, enabling emotional expression recognition.
  arXiv  Detail & Related papers  (2025-11-27T06:04:15Z)
• EmoSign: A Multimodal Dataset for Understanding Emotions in American Sign Language [26.020329504293777]
  EmoSign is the first sign video dataset containing sentiment and emotion labels for 200 American Sign Language (ASL) videos.<n>We also collect open-ended descriptions of emotion cues.
  arXiv  Detail & Related papers  (2025-05-20T22:14:38Z)
• Large Language Models Meet Contrastive Learning: Zero-Shot Emotion Recognition Across Languages [31.15696076055884]
  We propose leveraging contrastive learning to refine multilingual speech features and extend large language models.<n>Specifically, we employ a novel two-stage training framework to align speech signals with linguistic features in the emotional space.<n>To advance research in this field, we introduce a large-scale synthetic multilingual speech emotion dataset, M5SER.
  arXiv  Detail & Related papers  (2025-03-25T05:58:18Z)
• BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets for 28 Languages [93.92804151830744]
  We present BRIGHTER, a collection of multi-labeled, emotion-annotated datasets in 28 different languages.<n>We highlight the challenges related to the data collection and annotation processes.<n>We show that the BRIGHTER datasets represent a meaningful step towards addressing the gap in text-based emotion recognition.
  arXiv  Detail & Related papers  (2025-02-17T15:39:50Z)
• SHuBERT: Self-Supervised Sign Language Representation Learning via Multi-Stream Cluster Prediction [65.1590372072555]
  SHuBERT (Sign Hidden-Unit BERT) is a self-supervised contextual representation model learned from 1,000 hours of American Sign Language video.<n>SHuBERT adapts masked token prediction objectives to multi-stream visual sign language input, learning to predict multiple targets corresponding to clustered hand, face, and body pose streams.<n>SHuBERT achieves state-of-the-art performance across multiple tasks including sign language translation, isolated sign language recognition, and fingerspelling detection.
  arXiv  Detail & Related papers  (2024-11-25T03:13:08Z)
• Classification in Japanese Sign Language Based on Dynamic Facial Expressions [0.0]
  Research on Japanese Sign Language (JSL) recognition is limited due to a lack of datasets.<n>In JSL, sentence types such as affirmative statements and questions are distinguished by facial expressions.<n>Our proposed method utilizes a neural network to analyze facial features and classify sentence types.
  arXiv  Detail & Related papers  (2024-11-10T03:34:34Z)
• Scaling up Multimodal Pre-training for Sign Language Understanding [96.17753464544604]
  Sign language serves as the primary meaning of communication for the deaf-mute community. To facilitate communication between the deaf-mute and hearing people, a series of sign language understanding (SLU) tasks have been studied. These tasks investigate sign language topics from diverse perspectives and raise challenges in learning effective representation of sign language videos.
  arXiv  Detail & Related papers  (2024-08-16T06:04:25Z)
• Self-Supervised Representation Learning with Spatial-Temporal Consistency for Sign Language Recognition [96.62264528407863]
  We propose a self-supervised contrastive learning framework to excavate rich context via spatial-temporal consistency. Inspired by the complementary property of motion and joint modalities, we first introduce first-order motion information into sign language modeling. Our method is evaluated with extensive experiments on four public benchmarks, and achieves new state-of-the-art performance with a notable margin.
  arXiv  Detail & Related papers  (2024-06-15T04:50:19Z)
• Improving Continuous Sign Language Recognition with Cross-Lingual Signs [29.077175863743484]
  We study the feasibility of utilizing multilingual sign language corpora to facilitate continuous sign language recognition. We first build two sign language dictionaries containing isolated signs that appear in two datasets. Then we identify the sign-to-sign mappings between two sign languages via a well-optimized isolated sign language recognition model.
  arXiv  Detail & Related papers  (2023-08-21T15:58:47Z)
• Skeleton Based Sign Language Recognition Using Whole-body Keypoints [71.97020373520922]
  Sign language is used by deaf or speech impaired people to communicate. Skeleton-based recognition is becoming popular that it can be further ensembled with RGB-D based method to achieve state-of-the-art performance. Inspired by the recent development of whole-body pose estimation citejin 2020whole, we propose recognizing sign language based on the whole-body key points and features.
  arXiv  Detail & Related papers  (2021-03-16T03:38:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.