American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM

• URL: http://arxiv.org/abs/2409.10874v1
• Date: Tue, 17 Sep 2024 04:00:33 GMT
• Title: American Sign Language to Text Translation using Transformer and Seq2Seq with LSTM
• Authors: Gregorius Guntur Sunardi Putra, Adifa Widyadhani Chanda D'Layla, Dimas Wahono, Riyanarto Sarno, Agus Tri Haryono,
• Abstract summary: American Sign Language is one of the sign languages used. Development of neural machine translation technology is moving towards sign language translation. Transformer became the state-of-the-art in natural language processing.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sign language translation is one of the important issues in communication between deaf and hearing people, as it expresses words through hand, body, and mouth movements. American Sign Language is one of the sign languages used, one of which is the alphabetic sign. The development of neural machine translation technology is moving towards sign language translation. Transformer became the state-of-the-art in natural language processing. This study compares the Transformer with the Sequence-to-Sequence (Seq2Seq) model in translating sign language to text. In addition, an experiment was conducted by adding Residual Long Short-Term Memory (ResidualLSTM) in the Transformer. The addition of ResidualLSTM to the Transformer reduces the performance of the Transformer model by 23.37% based on the BLEU Score value. In comparison, the Transformer itself increases the BLEU Score value by 28.14 compared to the Seq2Seq model.

Related papers

• Enhanced Transformer Architecture for Natural Language Processing [2.6071653283020915]
  Transformer is a state-of-the-art model in the field of natural language processing (NLP) In this paper, a novel structure of Transformer is proposed. It is featured by full layer normalization, weighted residual connection, positional encoding exploiting reinforcement learning, and zero masked self-attention. The proposed Transformer model, which is called Enhanced Transformer, is validated by the bilingual evaluation understudy (BLEU) score obtained with the Multi30k translation dataset.
  arXiv  Detail & Related papers  (2023-10-17T01:59:07Z)
• Is context all you need? Scaling Neural Sign Language Translation to Large Domains of Discourse [34.70927441846784]
  Sign Language Translation (SLT) is a challenging task that aims to generate spoken language sentences from sign language videos. We propose a novel multi-modal transformer architecture that tackles the translation task in a context-aware manner, as a human would. We report significant improvements on state-of-the-art translation performance using contextual information, nearly doubling the reported BLEU-4 scores of baseline approaches.
  arXiv  Detail & Related papers  (2023-08-18T15:27:22Z)
• Code-Switching Text Generation and Injection in Mandarin-English ASR [57.57570417273262]
  We investigate text generation and injection for improving the performance of an industry commonly-used streaming model, Transformer-Transducer (T-T) We first propose a strategy to generate code-switching text data and then investigate injecting generated text into T-T model explicitly by Text-To-Speech (TTS) conversion or implicitly by tying speech and text latent spaces. Experimental results on the T-T model trained with a dataset containing 1,800 hours of real Mandarin-English code-switched speech show that our approaches to inject generated code-switching text significantly boost the performance of T-T models.
  arXiv  Detail & Related papers  (2023-03-20T09:13:27Z)
• GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation [107.2752114891855]
  Transformer structure, stacked by a sequence of encoder and decoder network layers, achieves significant development in neural machine translation. We propose the Group-Transformer model (GTrans) that flexibly divides multi-layer representations of both encoder and decoder into different groups and then fuses these group features to generate target words.
  arXiv  Detail & Related papers  (2022-07-29T04:10:36Z)
• Sentence Bottleneck Autoencoders from Transformer Language Models [53.350633961266375]
  We build a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
  arXiv  Detail & Related papers  (2021-08-31T19:39:55Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• Segatron: Segment-Aware Transformer for Language Modeling and Understanding [79.84562707201323]
  We propose a segment-aware Transformer (Segatron) to generate better contextual representations from sequential tokens. We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset.
  arXiv  Detail & Related papers  (2020-04-30T17:38:27Z)
• Character-Level Translation with Self-attention [9.864260997723974]
  We explore the suitability of self-attention models for character-level neural machine translation. We test the standard transformer model and a novel variant in which the encoder block combines information from nearby characters using convolutions. Our transformer variant consistently outperforms the standard transformer at the character-level and converges faster while learning more robust character-level alignments.
  arXiv  Detail & Related papers  (2020-04-30T14:05:26Z)
• Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation [59.38247587308604]
  We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T dataset. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models.
  arXiv  Detail & Related papers  (2020-03-30T21:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.