Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding

• URL: http://arxiv.org/abs/2102.06750v1
• Date: Fri, 12 Feb 2021 20:09:08 GMT
• Title: Do as I mean, not as I say: Sequence Loss Training for Spoken Language Understanding
• Authors: Milind Rao, Pranav Dheram, Gautam Tiwari, Anirudh Raju, Jasha Droppo, Ariya Rastrow, Andreas Stolcke
• Abstract summary: Spoken language understanding (SLU) systems extract transcriptions, as well as semantics of intent or named entities from speech. We propose non-differentiable sequence losses based on SLU metrics as a proxy for semantic error and use the REINFORCE trick to train ASR and SLU models with this loss. We show that custom sequence loss training is the state-of-the-art on open SLU datasets and leads to 6% relative improvement in both ASR and NLU performance metrics.
• Score: 22.652754839140744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spoken language understanding (SLU) systems extract transcriptions, as well as semantics of intent or named entities from speech, and are essential components of voice activated systems. SLU models, which either directly extract semantics from audio or are composed of pipelined automatic speech recognition (ASR) and natural language understanding (NLU) models, are typically trained via differentiable cross-entropy losses, even when the relevant performance metrics of interest are word or semantic error rates. In this work, we propose non-differentiable sequence losses based on SLU metrics as a proxy for semantic error and use the REINFORCE trick to train ASR and SLU models with this loss. We show that custom sequence loss training is the state-of-the-art on open SLU datasets and leads to 6% relative improvement in both ASR and NLU performance metrics on large proprietary datasets. We also demonstrate how the semantic sequence loss training paradigm can be used to update ASR and SLU models without transcripts, using semantic feedback alone.

Related papers

• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding [55.39105863825107]
  We propose Mutual Learning and Large-Margin Contrastive Learning (ML-LMCL) to improve automatic speech recognition (ASR) robustness. In fine-tuning, we apply mutual learning and train two SLU models on the manual transcripts and the ASR transcripts, respectively. Experiments on three datasets show that ML-LMCL outperforms existing models and achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-11-19T16:53:35Z)
• Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target [58.59044226658916]
  Spoken Language Understanding (SLU) is a task that aims to extract semantic information from spoken utterances. We propose to use discrete units as intermediate guidance to improve textless SLU performance.
  arXiv  Detail & Related papers  (2023-05-29T14:00:24Z)
• End-to-end spoken language understanding using joint CTC loss and self-supervised, pretrained acoustic encoders [13.722028186368737]
  We leverage self-supervised acoustic encoders fine-tuned with Connectionist Temporal Classification to extract textual embeddings. Our model achieves 4% absolute improvement over the the state-of-the-art (SOTA) dialogue act classification model on the DSTC2 dataset.
  arXiv  Detail & Related papers  (2023-05-04T15:36:37Z)
• Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models [94.30953696090758]
  We build compositional end-to-end spoken language understanding systems. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to token-level representations. Our models outperform both cascaded and direct end-to-end models on a labeling task of named entity recognition.
  arXiv  Detail & Related papers  (2022-10-27T19:33:18Z)
• STOP: A dataset for Spoken Task Oriented Semantic Parsing [66.14615249745448]
  End-to-end spoken language understanding (SLU) predicts intent directly from audio using a single model. We release the Spoken Task-Oriented semantic Parsing (STOP) dataset, the largest and most complex SLU dataset to be publicly available. In addition to the human-recorded audio, we are releasing a TTS-generated version to benchmark the performance for low-resource domain adaptation of end-to-end SLU systems.
  arXiv  Detail & Related papers  (2022-06-29T00:36:34Z)
• Deliberation Model for On-Device Spoken Language Understanding [69.5587671262691]
  We propose a novel deliberation-based approach to end-to-end (E2E) spoken language understanding (SLU) We show that our approach can significantly reduce the degradation when moving from natural speech to synthetic speech training.
  arXiv  Detail & Related papers  (2022-04-04T23:48:01Z)
• End-to-End Spoken Language Understanding using RNN-Transducer ASR [14.267028645397266]
  We propose an end-to-end trained spoken language understanding (SLU) system that extracts transcripts, intents and slots from an input speech utterance. It consists of a streaming recurrent neural network transducer (RNNT) based automatic speech recognition (ASR) model connected to a neural natural language understanding (NLU) model through a neural interface.
  arXiv  Detail & Related papers  (2021-06-30T09:20:32Z)
• Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces [17.030832205343195]
  We consider the problem of spoken language understanding (SLU) of extracting natural language intents from speech directed at voice assistants. An end-to-end joint SLU model can be built to a required specification opening up the opportunity to deploy on hardware constrained scenarios. We show that the jointly trained model shows improvements to ASR incorporating semantic information from NLU and also improves NLU by exposing it to ASR confusion encoded in the hidden layer.
  arXiv  Detail & Related papers  (2020-08-14T02:43:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.