Emotion Recognition in Sign Language Conversation

• URL: http://arxiv.org/abs/2605.23328v1
• Date: Fri, 22 May 2026 07:44:20 GMT
• Title: Emotion Recognition in Sign Language Conversation
• Abstract summary: We introduce the ERC task to sign language video analysis and propose the eJSL Dialog dataset.<n>The dataset contains 1,920 video samples organized into 480 unique dialogues.<n>We conduct systematic benchmarking on this dataset using models ranging from isolated visual networks to multimodal conversational architectures.
• Score: 9.580297565885376
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emotion Recognition in Conversation is a core component of affective computing, while current resources of sign language emotion datasets primarily focus on isolated sentences and lack conversational context. Models trained exclusively on these isolated utterances demonstrate degraded performance in real world scenarios because they cannot utilize historical dialogue flow. To address this structural limitation, we introduce the ERC task to sign language video analysis and propose the eJSL Dialog dataset. Constructed using the scripts from the STUDIES corpus, the dataset contains 1,920 video samples organized into 480 unique dialogues. We conduct systematic benchmarking on this dataset using models ranging from isolated visual networks to multimodal conversational architectures. The results reveal a domain gap when applying generic multimodal conversational emotion recognition models to sign language. These findings demonstrate the explicit need for context aware visual extractors specific to sign language and indicate that expanding the scale of conversational datasets to support large scale pre-training is a necessary next step for future research.

Related papers

• Towards Continuous Sign Language Conversation from Isolated Signs [15.139358499214529]
  We introduce SignaVox-W, the largest labeled isolated-sign vocabulary to date, and SignaVox-U, a continuous 3D sign conversation dataset.<n>With the resulting data, we train SignaVox, a direct sign-to-sign conversational model that generates 3D body, hand, and facial motion responses from prior signing context.
  arXiv  Detail & Related papers  (2026-05-14T11:22:27Z)
• Emotion Transcription in Conversation: A Benchmark for Capturing Subtle and Complex Emotional States through Natural Language [3.873439892819606]
  We propose a novel task named Emotion Transcription in Conversation (ETC)<n>This task focuses on generating natural language descriptions that accurately reflect speakers' emotional states within conversational contexts.<n>We constructed a Japanese dataset comprising text-based dialogues annotated with participants' self-reported emotional states, described in natural language.
  arXiv  Detail & Related papers  (2026-03-07T10:14:05Z)
• Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models [3.8673630752805446]
  The aim is to investigate the extent to which the linguistic context alone can inform the detection of mentions.<n>We adapt a pretrained large language model (LLM) to perform a relatively course-grained annotation of mention spans in unfolding conversations.<n>Our findings indicate that even when using a moderately sized LLM, relatively small datasets, and parameter-efficient fine-tuning, a text-only approach can be effective.
  arXiv  Detail & Related papers  (2025-06-26T14:14:20Z)
• Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues [19.675409379345172]
  We introduce MARS, a multimodal language model designed to understand and generate nonverbal cues alongside text.<n>Our key innovation is VENUS, a large-scale dataset comprising annotated videos with time-aligned text, facial expressions, and body language.
  arXiv  Detail & Related papers  (2025-06-01T11:07:25Z)
• Akan Cinematic Emotions (ACE): A Multimodal Multi-party Dataset for Emotion Recognition in Movie Dialogues [4.894647740789939]
  The Akan Conversation Emotion dataset is the first multimodal emotion dialogue dataset for an African language.<n>It contains 385 emotion-labeled dialogues and 6,162 utterances across audio, visual, and textual modalities.<n>The presence of prosodic labels in this dataset also makes it the first prosodically annotated African language dataset.
  arXiv  Detail & Related papers  (2025-02-16T03:24:33Z)
• SER_AMPEL: a multi-source dataset for speech emotion recognition of Italian older adults [58.49386651361823]
  SER_AMPEL is a multi-source dataset for speech emotion recognition (SER) It is collected with the aim of providing a reference for speech emotion recognition in case of Italian older adults. The evidence of the need for such a dataset emerges from the analysis of the state of the art.
  arXiv  Detail & Related papers  (2023-11-24T13:47:25Z)
• Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension [81.47133615169203]
  We propose compositional learning for holistic interaction across utterances beyond the sequential contextualization from PrLMs. We employ domain-adaptive training strategies to help the model adapt to the dialogue domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-10T13:18:25Z)
• Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling [80.51094098799736]
  We propose Bidirectional Information Decoupling Network (BiDeN) as a universal dialogue encoder. BiDeN explicitly incorporates both the past and future contexts and can be generalized to a wide range of dialogue-related tasks. Experimental results on datasets of different downstream tasks demonstrate the universality and effectiveness of our BiDeN.
  arXiv  Detail & Related papers  (2022-04-18T03:51:46Z)
• Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation [70.81596088969378]
  Cross-lingual Outline-based Dialogue dataset (termed COD) enables natural language understanding. COD enables dialogue state tracking, and end-to-end dialogue modelling and evaluation in 4 diverse languages.
  arXiv  Detail & Related papers  (2022-01-31T18:11:21Z)
• $C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues [97.25466640240619]
  Video-grounded dialogue systems aim to integrate video understanding and dialogue understanding to generate responses relevant to both the dialogue and video context. Most existing approaches employ deep learning models and have achieved remarkable performance, given the relatively small datasets available. We propose a novel approach of Compositional Counterfactual Contrastive Learning to develop contrastive training between factual and counterfactual samples in video-grounded dialogues.
  arXiv  Detail & Related papers  (2021-06-16T16:05:27Z)
• Dialogue History Matters! Personalized Response Selectionin Multi-turn Retrieval-based Chatbots [62.295373408415365]
  We propose a personalized hybrid matching network (PHMN) for context-response matching. Our contributions are two-fold: 1) our model extracts personalized wording behaviors from user-specific dialogue history as extra matching information. We evaluate our model on two large datasets with user identification, i.e., personalized dialogue Corpus Ubuntu (P- Ubuntu) and personalized Weibo dataset (P-Weibo)
  arXiv  Detail & Related papers  (2021-03-17T09:42:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.