Machine Translation from Signed to Spoken Languages: State of the Art and Challenges

• URL: http://arxiv.org/abs/2202.03086v4
• Date: Wed, 5 Apr 2023 06:30:14 GMT
• Title: Machine Translation from Signed to Spoken Languages: State of the Art and Challenges
• Authors: Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, Joni Dambre
• Abstract summary: We give a high-level introduction to sign language linguistics and machine translation. We find that significant advances have been made on the shoulders of spoken language machine translation research. We advocate for interdisciplinary research and to base future research on linguistic analysis of sign languages.
• Score: 9.292669129832605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic translation from signed to spoken languages is an interdisciplinary research domain, lying on the intersection of computer vision, machine translation and linguistics. Nevertheless, research in this domain is performed mostly by computer scientists in isolation. As the domain is becoming increasingly popular - the majority of scientific papers on the topic of sign language translation have been published in the past three years - we provide an overview of the state of the art as well as some required background in the different related disciplines. We give a high-level introduction to sign language linguistics and machine translation to illustrate the requirements of automatic sign language translation. We present a systematic literature review to illustrate the state of the art in the domain and then, harking back to the requirements, lay out several challenges for future research. We find that significant advances have been made on the shoulders of spoken language machine translation research. However, current approaches are often not linguistically motivated or are not adapted to the different input modality of sign languages. We explore challenges related to the representation of sign language data, the collection of datasets, the need for interdisciplinary research and requirements for moving beyond research, towards applications. Based on our findings, we advocate for interdisciplinary research and to base future research on linguistic analysis of sign languages. Furthermore, the inclusion of deaf and hearing end users of sign language translation applications in use case identification, data collection and evaluation is of the utmost importance in the creation of useful sign language translation models. We recommend iterative, human-in-the-loop, design and development of sign language translation models.

Related papers

• From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation [0.0]
  There is a pressing need for an efficient, signs-driven, integrated end-to-end translation system. We present the requirements of a real-time Quality-of-Service sign language ma-chine translation system underpinned by accurate deep learning algorithms.
  arXiv  Detail & Related papers  (2024-08-27T07:11:45Z)
• Understanding Cross-Lingual Alignment -- A Survey [52.572071017877704]
  Cross-lingual alignment is the meaningful similarity of representations across languages in multilingual language models. We survey the literature of techniques to improve cross-lingual alignment, providing a taxonomy of methods and summarising insights from throughout the field.
  arXiv  Detail & Related papers  (2024-04-09T11:39:53Z)
• Systemic Biases in Sign Language AI Research: A Deaf-Led Call to Reevaluate Research Agendas [1.9285000127136378]
  We conduct a systematic review of 101 recent papers in sign language AI. We identify significant biases in the current state of sign language AI research. We take the position that the field lacks meaningful input from Deaf stakeholders.
  arXiv  Detail & Related papers  (2024-03-05T00:37:36Z)
• JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing [2.9936326613596775]
  JWSign dataset consists of 2,530 hours of Bible translations in 98 sign languages. We train multilingual systems, including some that take into account the typological relatedness of signed or spoken languages.
  arXiv  Detail & Related papers  (2023-11-16T20:02:44Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)
• Including Signed Languages in Natural Language Processing [48.62744923724317]
  Signed languages are the primary means of communication for many deaf and hard of hearing individuals. This position paper calls on the NLP community to include signed languages as a research area with high social and scientific impact.
  arXiv  Detail & Related papers  (2021-05-11T17:37:55Z)
• Crossing the Conversational Chasm: A Primer on Multilingual Task-Oriented Dialogue Systems [51.328224222640614]
  Current state-of-the-art ToD models based on large pretrained neural language models are data hungry. Data acquisition for ToD use cases is expensive and tedious.
  arXiv  Detail & Related papers  (2021-04-17T15:19:56Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)
• Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation [59.38247587308604]
  We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T dataset. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models.
  arXiv  Detail & Related papers  (2020-03-30T21:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.