Considerations for meaningful sign language machine translation based on glosses

• URL: http://arxiv.org/abs/2211.15464v1
• Date: Mon, 28 Nov 2022 15:51:58 GMT
• Title: Considerations for meaningful sign language machine translation based on glosses
• Authors: Mathias M\"uller, Zifan Jiang, Amit Moryossef, Annette Rios, Sarah Ebling
• Abstract summary: In machine translation (MT), sign language translation based on glosses is a prominent approach. We find that limitations of glosses in general and limitations of specific datasets are not discussed in a transparent manner. We put forward concrete recommendations for future research on gloss translation.
• Score: 6.422262171968398
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic sign language processing is gaining popularity in Natural Language Processing (NLP) research (Yin et al., 2021). In machine translation (MT) in particular, sign language translation based on glosses is a prominent approach. In this paper, we review recent works on neural gloss translation. We find that limitations of glosses in general and limitations of specific datasets are not discussed in a transparent manner and that there is no common standard for evaluation. To address these issues, we put forward concrete recommendations for future research on gloss translation. Our suggestions advocate awareness of the inherent limitations of gloss-based approaches, realistic datasets, stronger baselines and convincing evaluation.

Related papers

• Universal Gloss-level Representation for Gloss-free Sign Language Translation and Production [9.065171626657818]
  Universal Gloss-level Representation (UniGloR) is a unified and self-supervised solution for both Sign Language Translation and Sign Language Production. Our results demonstrate UniGloR's effectiveness in the translation and production tasks. Our study suggests that self-supervised learning can be made in a unified manner, paving the way for innovative and practical applications.
  arXiv  Detail & Related papers  (2024-07-03T07:12:36Z)
• Gloss2Text: Sign Language Gloss translation using LLMs and Semantically Aware Label Smoothing [21.183453511034767]
  We propose several advances by leveraging pre-trained large language models (LLMs), data augmentation, and novel label-smoothing loss function. Our approach surpasses state-of-the-art performance in em Gloss2Text translation.
  arXiv  Detail & Related papers  (2024-07-01T15:46:45Z)
• Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining [56.26550923909137]
  Gloss-Free Sign Language Translation (SLT) is a challenging task due to its cross-domain nature. We propose a novel Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-) Our approach involves two stages: (i) integrating Contrastive Language-Image Pre-training with masked self-supervised learning to create pre-tasks that bridge the semantic gap between visual and textual representations and restore masked sentences, and (ii) constructing an end-to-end architecture with an encoder-decoder-like structure that inherits the parameters of the pre-trained Visual and Text Decoder from
  arXiv  Detail & Related papers  (2023-07-27T10:59:18Z)
• Gloss-Free End-to-End Sign Language Translation [59.28829048788345]
  We design the Gloss-Free End-to-end sign language translation framework (GloFE) Our method improves the performance of SLT in the gloss-free setting by exploiting the shared underlying semantics of signs and the corresponding spoken translation. We obtained state-of-the-art results on large-scale datasets, including OpenASL and How2Sign.
  arXiv  Detail & Related papers  (2023-05-22T09:57:43Z)
• Keypoint based Sign Language Translation without Glosses [7.240731862549344]
  We propose a new keypoint normalization method for performing translation based on the skeleton point of the signer. It contributed to performance improvement by a customized normalization method depending on the body parts. Our method can be applied to various datasets in a way that can be applied to datasets without glosses.
  arXiv  Detail & Related papers  (2022-04-22T05:37:56Z)
• A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translation [54.29679610921429]
  Existing sign language datasets contain only about 10K-20K pairs of sign videos, gloss annotations and texts. Data is thus a bottleneck for training effective sign language translation models. This simple baseline surpasses the previous state-of-the-art results on two sign language translation benchmarks.
  arXiv  Detail & Related papers  (2022-03-08T18:59:56Z)
• SimulSLT: End-to-End Simultaneous Sign Language Translation [55.54237194555432]
  Existing sign language translation methods need to read all the videos before starting the translation. We propose SimulSLT, the first end-to-end simultaneous sign language translation model. SimulSLT achieves BLEU scores that exceed the latest end-to-end non-simultaneous sign language translation model.
  arXiv  Detail & Related papers  (2021-12-08T11:04:52Z)
• Improving Sign Language Translation with Monolingual Data by Sign Back-Translation [105.83166521438463]
  We propose a sign back-translation (SignBT) approach, which incorporates massive spoken language texts into sign training. With a text-to-gloss translation model, we first back-translate the monolingual text to its gloss sequence. Then, the paired sign sequence is generated by splicing pieces from an estimated gloss-to-sign bank at the feature level.
  arXiv  Detail & Related papers  (2021-05-26T08:49:30Z)
• Data Augmentation for Sign Language Gloss Translation [115.13684506803529]
  Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss-totext translation. We focus here on gloss-to-text translation, which we treat as a low-resource neural machine translation (NMT) problem. By pre-training on the thus obtained synthetic data, we improve translation from American Sign Language (ASL) to English and German Sign Language (DGS) to German by up to 3.14 and 2.20 BLEU, respectively.
  arXiv  Detail & Related papers  (2021-05-16T16:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.