RescoreBERT: Discriminative Speech Recognition Rescoring with BERT

• URL: http://arxiv.org/abs/2202.01094v1
• Date: Wed, 2 Feb 2022 15:45:26 GMT
• Title: RescoreBERT: Discriminative Speech Recognition Rescoring with BERT
• Authors: Liyan Xu, Yile Gu, Jari Kolehmainen, Haidar Khan, Ankur Gandhe, Ariya Rastrow, Andreas Stolcke, Ivan Bulyko
• Abstract summary: We show how to train a BERT-based rescoring model with MWER loss, to incorporate the improvements of a discriminative loss into fine-tuning of deep bidirectional pretrained models for ASR. We name this approach RescoreBERT, and evaluate it on the LibriSpeech corpus, and it reduces WER by 6.6%/3.4% relative on clean/other test sets over a BERT baseline without discriminative objective.
• Score: 21.763672436079872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Second-pass rescoring is an important component in automatic speech recognition (ASR) systems that is used to improve the outputs from a first-pass decoder by implementing a lattice rescoring or $n$-best re-ranking. While pretraining with a masked language model (MLM) objective has received great success in various natural language understanding (NLU) tasks, it has not gained traction as a rescoring model for ASR. Specifically, training a bidirectional model like BERT on a discriminative objective such as minimum WER (MWER) has not been explored. Here we where show how to train a BERT-based rescoring model with MWER loss, to incorporate the improvements of a discriminative loss into fine-tuning of deep bidirectional pretrained models for ASR. We propose a fusion strategy that incorporates the MLM into the discriminative training process to effectively distill the knowledge from a pretrained model. We further propose an alternative discriminative loss. We name this approach RescoreBERT, and evaluate it on the LibriSpeech corpus, and it reduces WER by 6.6%/3.4% relative on clean/other test sets over a BERT baseline without discriminative objective. We also evaluate our method on an internal dataset from a conversational agent and find that it reduces both latency and WER (by 3-8% relative) over an LSTM rescoring model.

Related papers

• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Zero-Shot Automatic Pronunciation Assessment [19.971348810774046]
  We propose a novel zero-shot APA method based on the pre-trained acoustic model, HuBERT. Experimental results on speechocean762 demonstrate that the proposed method achieves comparable performance to supervised regression baselines.
  arXiv  Detail & Related papers  (2023-05-31T05:17:17Z)
• Integrate Lattice-Free MMI into End-to-End Speech Recognition [87.01137882072322]
  In automatic speech recognition (ASR) research, discriminative criteria have achieved superior performance in DNN-HMM systems. With this motivation, the adoption of discriminative criteria is promising to boost the performance of end-to-end (E2E) ASR systems. Previous works have introduced the minimum Bayesian risk (MBR, one of the discriminative criteria) into E2E ASR systems. In this work, novel algorithms are proposed in this work to integrate another widely used discriminative criterion, lattice-free maximum mutual information (LF-MMI) into E2E
  arXiv  Detail & Related papers  (2022-03-29T14:32:46Z)
• Sequence-level self-learning with multiple hypotheses [53.04725240411895]
  We develop new self-learning techniques with an attention-based sequence-to-sequence (seq2seq) model for automatic speech recognition (ASR) In contrast to conventional unsupervised learning approaches, we adopt the emphmulti-task learning (MTL) framework. Our experiment results show that our method can reduce the WER on the British speech data from 14.55% to 10.36% compared to the baseline model trained with the US English data only.
  arXiv  Detail & Related papers  (2021-12-10T20:47:58Z)
• A Method to Reveal Speaker Identity in Distributed ASR Training, and How to Counter It [3.18475216176047]
  We design the first method for revealing the identity of the speaker of a training utterance with access only to a gradient. We show that it is possible to reveal the speaker's identity with 34% top-1 accuracy (51% top-5 accuracy) on the LibriSpeech dataset.
  arXiv  Detail & Related papers  (2021-04-15T23:15:12Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• Incremental Learning for End-to-End Automatic Speech Recognition [41.297106772785206]
  We propose an incremental learning method for end-to-end Automatic Speech Recognition (ASR) We design a novel explainability-based knowledge distillation for ASR models, which is combined with a response-based knowledge distillation to maintain the original model's predictions and the "reason" for the predictions. Results on a multi-stage sequential training task show that our method outperforms existing ones in mitigating forgetting.
  arXiv  Detail & Related papers  (2020-05-11T08:18:08Z)
• Joint Contextual Modeling for ASR Correction and Language Understanding [60.230013453699975]
  We propose multi-task neural approaches to perform contextual language correction on ASR outputs jointly with language understanding (LU) We show that the error rates of off the shelf ASR and following LU systems can be reduced significantly by 14% relative with joint models trained using small amounts of in-domain data.
  arXiv  Detail & Related papers  (2020-01-28T22:09:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.