Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding

• URL: http://arxiv.org/abs/2204.04965v1
• Date: Mon, 11 Apr 2022 09:30:08 GMT
• Title: Multistream neural architectures for cued-speech recognition using a pre-trained visual feature extractor and constrained CTC decoding
• Authors: Sanjana Sankar (GIPSA-CRISSP), Denis Beautemps (GIPSA-CRISSP), Thomas Hueber (GIPSA-CRISSP)
• Abstract summary: Cued Speech (CS) is a visual communication tool that helps people with hearing impairment to understand spoken language. The proposed approach is based on a pre-trained hand and lips tracker used for visual feature extraction and a phonetic decoder based on a multistream recurrent neural network. With a decoding accuracy at the phonetic level of 70.88%, the proposed system outperforms our previous CNN-HMM decoder and competes with more complex baselines.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes a simple and effective approach for automatic recognition of Cued Speech (CS), a visual communication tool that helps people with hearing impairment to understand spoken language with the help of hand gestures that can uniquely identify the uttered phonemes in complement to lipreading. The proposed approach is based on a pre-trained hand and lips tracker used for visual feature extraction and a phonetic decoder based on a multistream recurrent neural network trained with connectionist temporal classification loss and combined with a pronunciation lexicon. The proposed system is evaluated on an updated version of the French CS dataset CSF18 for which the phonetic transcription has been manually checked and corrected. With a decoding accuracy at the phonetic level of 70.88%, the proposed system outperforms our previous CNN-HMM decoder and competes with more complex baselines.

Related papers

• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• Contextual-Utterance Training for Automatic Speech Recognition [65.4571135368178]
  We propose a contextual-utterance training technique which makes use of the previous and future contextual utterances. Also, we propose a dual-mode contextual-utterance training technique for streaming automatic speech recognition (ASR) systems. The proposed technique is able to reduce both the WER and the average last token emission latency by more than 6% and 40ms relative.
  arXiv  Detail & Related papers  (2022-10-27T08:10:44Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• Audio-visual multi-channel speech separation, dereverberation and recognition [70.34433820322323]
  This paper proposes an audio-visual multi-channel speech separation, dereverberation and recognition approach. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches. Experiments conducted on the LRS2 dataset suggest that the proposed audio-visual multi-channel speech separation, dereverberation and recognition system outperforms the baseline.
  arXiv  Detail & Related papers  (2022-04-05T04:16:03Z)
• Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizers [13.372686722688325]
  Training of end-to-end speech recognizers always requires transcribed utterances. This paper proposes a method for alleviating this issue by transferring knowledge from a language model neural network that can be pretrained with text-only data.
  arXiv  Detail & Related papers  (2022-02-16T07:02:24Z)
• Speech recognition for air traffic control via feature learning and end-to-end training [8.755785876395363]
  We propose a new automatic speech recognition (ASR) system based on feature learning and an end-to-end training procedure for air traffic control (ATC) systems. The proposed model integrates the feature learning block, recurrent neural network (RNN), and connectionist temporal classification loss. Thanks to the ability to learn representations from raw waveforms, the proposed model can be optimized in a complete end-to-end manner.
  arXiv  Detail & Related papers  (2021-11-04T06:38:21Z)
• A review of on-device fully neural end-to-end automatic speech recognition algorithms [20.469868150587075]
  We review various end-to-end automatic speech recognition algorithms and their optimization techniques for on-device applications. fully neural network end-to-end speech recognition algorithms have been proposed. We extensively discuss their structures, performance, and advantages compared to conventional algorithms.
  arXiv  Detail & Related papers  (2020-12-14T22:18:08Z)
• AutoSpeech: Neural Architecture Search for Speaker Recognition [108.69505815793028]
  We propose the first neural architecture search approach approach for the speaker recognition tasks, named as AutoSpeech. Our algorithm first identifies the optimal operation combination in a neural cell and then derives a CNN model by stacking the neural cell for multiple times. Results demonstrate that the derived CNN architectures significantly outperform current speaker recognition systems based on VGG-M, ResNet-18, and ResNet-34 back-bones, while enjoying lower model complexity.
  arXiv  Detail & Related papers  (2020-05-07T02:53:47Z)
• End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection [48.80449801938696]
  This paper integrates a voice activity detection function with end-to-end automatic speech recognition. We focus on connectionist temporal classification ( CTC) and its extension ofsynchronous/attention. We use the labels as a cue for detecting speech segments with simple thresholding.
  arXiv  Detail & Related papers  (2020-02-03T03:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.