A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition

• URL: http://arxiv.org/abs/2002.11268v3
• Date: Fri, 28 Feb 2020 01:40:54 GMT
• Title: A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognition
• Authors: Erik McDermott, Hasim Sak, Ehsan Variani
• Abstract summary: This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR) An RNN-T ASR model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data.
• Score: 9.184319271887531
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This article describes a density ratio approach to integrating external Language Models (LMs) into end-to-end models for Automatic Speech Recognition (ASR). Applied to a Recurrent Neural Network Transducer (RNN-T) ASR model trained on a given domain, a matched in-domain RNN-LM, and a target domain RNN-LM, the proposed method uses Bayes' Rule to define RNN-T posteriors for the target domain, in a manner directly analogous to the classic hybrid model for ASR based on Deep Neural Networks (DNNs) or LSTMs in the Hidden Markov Model (HMM) framework (Bourlard & Morgan, 1994). The proposed approach is evaluated in cross-domain and limited-data scenarios, for which a significant amount of target domain text data is used for LM training, but only limited (or no) {audio, transcript} training data pairs are used to train the RNN-T. Specifically, an RNN-T model trained on paired audio & transcript data from YouTube is evaluated for its ability to generalize to Voice Search data. The Density Ratio method was found to consistently outperform the dominant approach to LM and end-to-end ASR integration, Shallow Fusion.

Related papers

• Mask The Bias: Improving Domain-Adaptive Generalization of CTC-based ASR with Internal Language Model Estimation [14.840612036671734]
  Internal language model estimation (ILME) has been proposed to mitigate this bias for autoregressive models. We propose a novel ILME technique for CTC-based ASR models. Our method iteratively masks the audio timesteps to estimate a pseudo log-likelihood of the internal LM.
  arXiv  Detail & Related papers  (2023-05-05T20:35:42Z)
• Return of the RNN: Residual Recurrent Networks for Invertible Sentence Embeddings [0.0]
  This study presents a novel model for invertible sentence embeddings using a residual recurrent network trained on an unsupervised encoding task. Rather than the probabilistic outputs common to neural machine translation models, our approach employs a regression-based output layer to reconstruct the input sequence's word vectors. The model achieves high accuracy and fast training with the ADAM, a significant finding given that RNNs typically require memory units, such as LSTMs, or second-order optimization methods.
  arXiv  Detail & Related papers  (2023-03-23T15:59:06Z)
• Low-Resource Music Genre Classification with Cross-Modal Neural Model Reprogramming [129.4950757742912]
  We introduce a novel method for leveraging pre-trained models for low-resource (music) classification based on the concept of Neural Model Reprogramming (NMR) NMR aims at re-purposing a pre-trained model from a source domain to a target domain by modifying the input of a frozen pre-trained model. Experimental results suggest that a neural model pre-trained on large-scale datasets can successfully perform music genre classification by using this reprogramming method.
  arXiv  Detail & Related papers  (2022-11-02T17:38:33Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)
• Adaptive Discounting of Implicit Language Models in RNN-Transducers [33.63456351411599]
  We show how a lightweight adaptive LM discounting technique can be used with any RNN-T architecture. We obtain up to 4% and 14% relative reductions in overall WER and rare word PER, respectively, on a conversational, code-mixed Hindi-English ASR task.
  arXiv  Detail & Related papers  (2022-02-21T08:44:56Z)
• On Addressing Practical Challenges for RNN-Transduce [72.72132048437751]
  We adapt a well-trained RNN-T model to a new domain without collecting the audio data. We obtain word-level confidence scores by utilizing several types of features calculated during decoding. The proposed time stamping method can get less than 50ms word timing difference on average.
  arXiv  Detail & Related papers  (2021-04-27T23:31:43Z)
• Fast Text-Only Domain Adaptation of RNN-Transducer Prediction Network [0.0]
  We show that RNN-transducer models can be effectively adapted to new domains using only small amounts of textual data. We show with multiple ASR evaluation tasks how this method can provide relative gains of 10-45% in target task WER.
  arXiv  Detail & Related papers  (2021-04-22T15:21:41Z)
• A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings [50.524054820564395]
  We propose a new unsupervised model for mapping a variable-duration speech segment to a fixed-dimensional representation. The resulting acoustic word embeddings can form the basis of search, discovery, and indexing systems for low- and zero-resource languages.
  arXiv  Detail & Related papers  (2020-12-03T19:24:42Z)
• Exploring Pre-training with Alignments for RNN Transducer based End-to-End Speech Recognition [39.497407288772386]
  recurrent neural network transducer (RNN-T) architecture has become an emerging trend in end-to-end automatic speech recognition research. In this work, we leverage external alignments to seed the RNN-T model. Two different pre-training solutions are explored, referred to as encoder pre-training, and whole-network pre-training respectively.
  arXiv  Detail & Related papers  (2020-05-01T19:00:57Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)
• Model Fusion via Optimal Transport [64.13185244219353]
  We present a layer-wise model fusion algorithm for neural networks. We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
  arXiv  Detail & Related papers  (2019-10-12T22:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.