On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers

• URL: http://arxiv.org/abs/2309.14130v2
• Date: Sat, 13 Apr 2024 08:06:37 GMT
• Title: On the Relation between Internal Language Model and Sequence Discriminative Training for Neural Transducers
• Authors: Zijian Yang, Wei Zhou, Ralf Schlüter, Hermann Ney,
• Abstract summary: Internal language model (ILM) subtraction has been widely applied to improve the performance of the RNN-Transducer. We show that sequence discriminative training has a strong correlation with ILM subtraction from both theoretical and empirical points of view.
• Score: 52.88268942796418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Internal language model (ILM) subtraction has been widely applied to improve the performance of the RNN-Transducer with external language model (LM) fusion for speech recognition. In this work, we show that sequence discriminative training has a strong correlation with ILM subtraction from both theoretical and empirical points of view. Theoretically, we derive that the global optimum of maximum mutual information (MMI) training shares a similar formula as ILM subtraction. Empirically, we show that ILM subtraction and sequence discriminative training achieve similar effects across a wide range of experiments on Librispeech, including both MMI and minimum Bayes risk (MBR) criteria, as well as neural transducers and LMs of both full and limited context. The benefit of ILM subtraction also becomes much smaller after sequence discriminative training. We also provide an in-depth study to show that sequence discriminative training has a minimal effect on the commonly used zero-encoder ILM estimation, but a joint effect on both encoder and prediction + joint network for posterior probability reshaping including both ILM and blank suppression.

Related papers

• FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models [50.331708897857574]
  We introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications. FactorLLM achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed.
  arXiv  Detail & Related papers  (2024-08-15T16:45:16Z)
• CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models [68.64605538559312]
  In this paper, we analyze the MLLM instruction tuning from both theoretical and empirical perspectives. Inspired by our findings, we propose a measurement to quantitatively evaluate the learning balance. In addition, we introduce an auxiliary loss regularization method to promote updating of the generation distribution of MLLMs.
  arXiv  Detail & Related papers  (2024-07-29T23:18:55Z)
• Large Language Models are Biased Reinforcement Learners [0.0]
  We show that large language models (LLMs) exhibit behavioral signatures of a relative value bias. Computational cognitive modeling reveals that LLM behavior is well-described by a simple RL algorithm.
  arXiv  Detail & Related papers  (2024-05-19T01:43:52Z)
• Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration [39.35476224845088]
  Large language models (LLMs) exhibit complementary strengths in various tasks, motivating the research of LLM ensembling. We propose a training-free ensemble framework DeePEn, fusing the informative probability distributions yielded by different LLMs at each decoding step.
  arXiv  Detail & Related papers  (2024-04-19T08:52:22Z)
• Investigating the Effect of Language Models in Sequence Discriminative Training for Neural Transducers [36.60689278751483]
  We investigate the effect of language models (LMs) with different context lengths and label units (phoneme vs. word) used in sequence discriminative training. Experimental results on Librispeech show that using the word-level LM in training outperforms the phoneme-level LM. Our results reveal the pivotal importance of the hypothesis space quality in sequence discriminative training.
  arXiv  Detail & Related papers  (2023-10-11T09:53:17Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)
• On Language Model Integration for RNN Transducer based Speech Recognition [49.84285563767935]
  We study various ILM correction-based LM integration methods formulated in a common RNN-T framework. We provide a decoding interpretation on two major reasons for performance improvement with ILM correction. We also propose an exact-ILM training framework by extending the proof given in the hybrid autoregressive transducer.
  arXiv  Detail & Related papers  (2021-10-13T16:30:46Z)
• Counterfactual Maximum Likelihood Estimation for Training Deep Networks [83.44219640437657]
  Deep learning models are prone to learning spurious correlations that should not be learned as predictive clues. We propose a causality-based training framework to reduce the spurious correlations caused by observable confounders. We conduct experiments on two real-world tasks: Natural Language Inference (NLI) and Image Captioning.
  arXiv  Detail & Related papers  (2021-06-07T17:47:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.