Related papers: Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language

Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language

URL: http://arxiv.org/abs/2505.13784v1
Date: Tue, 20 May 2025 00:15:51 GMT
Title: Transfer Learning from Visual Speech Recognition to Mouthing Recognition in German Sign Language
Authors: Dinh Nam Pham, Eleftherios Avramidis,
Abstract summary: This work directly classifies mouthing instances to their corresponding words in the spoken language.<n>We explore the potential of transfer learning from Visual Speech Recognition to mouthing recognition in German Sign Language.
Score: 5.92177182003275
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: Sign Language Recognition (SLR) systems primarily focus on manual gestures, but non-manual features such as mouth movements, specifically mouthing, provide valuable linguistic information. This work directly classifies mouthing instances to their corresponding words in the spoken language while exploring the potential of transfer learning from Visual Speech Recognition (VSR) to mouthing recognition in German Sign Language. We leverage three VSR datasets: one in English, one in German with unrelated words and one in German containing the same target words as the mouthing dataset, to investigate the impact of task similarity in this setting. Our results demonstrate that multi-task learning improves both mouthing recognition and VSR accuracy as well as model robustness, suggesting that mouthing recognition should be treated as a distinct but related task to VSR. This research contributes to the field of SLR by proposing knowledge transfer from VSR to SLR datasets with limited mouthing annotations.

Related papers

Enhancing Multilingual ASR for Unseen Languages via Language Embedding Modeling [50.62091603179394]
Whisper, one of the most advanced ASR models, handles 99 languages effectively.<n>However, Whisper struggles with unseen languages, those not included in its pre-training.<n>We propose methods that exploit these relationships to enhance ASR performance on unseen languages.
arXiv Detail & Related papers (2024-12-21T04:05:43Z)
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper [96.43501666278316]
This paper proposes a powerful Visual Speech Recognition (VSR) method for multiple languages. We employ a Whisper model which can conduct both language identification and audio-based speech recognition. By comparing the performances of VSR models trained on automatic labels and the human-annotated labels, we show that we can achieve similar VSR performance to that of human-annotated labels.
arXiv Detail & Related papers (2023-09-15T16:53:01Z)
MASR: Multi-label Aware Speech Representation [36.2978180342839]
We propose MASR, a Multi-label Aware Speech Representation learning framework. MASR enables the inclusion of multiple external knowledge sources to enhance the utilization of meta-data information. We show significant performance improvements for the MASR over other established benchmarks.
arXiv Detail & Related papers (2023-07-20T16:09:57Z)
Label Aware Speech Representation Learning For Language Identification [49.197215416945596]
We propose a novel framework of combining self-supervised representation learning with the language label information for the pre-training task. This framework, termed as Label Aware Speech Representation (LASR) learning, uses a triplet based objective function to incorporate language labels along with the self-supervised loss function.
arXiv Detail & Related papers (2023-06-07T12:14:16Z)
Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning [11.50011780498048]
This paper presents a novel metric learning approach to address the performance gap between normal and silent speech in visual speech recognition (VSR) We propose to leverage the shared literal content between normal and silent speech and present a metric learning approach based on visemes. Our evaluation demonstrates that our method improves the accuracy of silent VSR, even when limited training data is available.
arXiv Detail & Related papers (2023-05-23T16:20:46Z)
LAE: Language-Aware Encoder for Monolingual and Multilingual ASR [87.74794847245536]
A novel language-aware encoder (LAE) architecture is proposed to handle both situations by disentangling language-specific information. Experiments conducted on Mandarin-English code-switched speech suggest that the proposed LAE is capable of discriminating different languages in frame-level.
arXiv Detail & Related papers (2022-06-05T04:03:12Z)
Multilingual Speech Recognition using Knowledge Transfer across Learning Processes [15.927513451432946]
Experimental results reveal the best pre-training strategy resulting in 3.55% relative reduction in overall WER. A combination of LEAP and SSL yields 3.51% relative reduction in overall WER when using language ID.
arXiv Detail & Related papers (2021-10-15T07:50:27Z)
LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech [63.84741259993937]
Self-Supervised Learning (SSL) using huge unlabeled data has been successfully explored for image and natural language processing. Recent works also investigated SSL from speech. We propose LeBenchmark: a reproducible framework for assessing SSL from speech.
arXiv Detail & Related papers (2021-04-23T08:27:09Z)
Meta-Transfer Learning for Code-Switched Speech Recognition [72.84247387728999]
We propose a new learning method, meta-transfer learning, to transfer learn on a code-switched speech recognition system in a low-resource setting. Our model learns to recognize individual languages, and transfer them so as to better recognize mixed-language speech by conditioning the optimization on the code-switching data.
arXiv Detail & Related papers (2020-04-29T14:27:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.