Written Term Detection Improves Spoken Term Detection

• URL: http://arxiv.org/abs/2407.04601v1
• Date: Fri, 5 Jul 2024 15:50:47 GMT
• Title: Written Term Detection Improves Spoken Term Detection
• Authors: Bolaji Yusuf, Murat Saraçlar,
• Abstract summary: We propose a multitask training objective which allows unpaired text to be integrated into E2E KWS without complicating indexing and search. In addition to training an E2E KWS model to retrieve text queries from spoken documents, we jointly train it to retrieve text queries from masked written documents. We show that this approach can effectively leverage unpaired text for KWS, with significant improvements in search performance across a wide variety of languages.
• Score: 9.961529254621432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end (E2E) approaches to keyword search (KWS) are considerably simpler in terms of training and indexing complexity when compared to approaches which use the output of automatic speech recognition (ASR) systems. This simplification however has drawbacks due to the loss of modularity. In particular, where ASR-based KWS systems can benefit from external unpaired text via a language model, current formulations of E2E KWS systems have no such mechanism. Therefore, in this paper, we propose a multitask training objective which allows unpaired text to be integrated into E2E KWS without complicating indexing and search. In addition to training an E2E KWS model to retrieve text queries from spoken documents, we jointly train it to retrieve text queries from masked written documents. We show empirically that this approach can effectively leverage unpaired text for KWS, with significant improvements in search performance across a wide variety of languages. We conduct analysis which indicates that these improvements are achieved because the proposed method improves document representations for words in the unpaired text. Finally, we show that the proposed method can be used for domain adaptation in settings where in-domain paired data is scarce or nonexistent.

Related papers

• Learning Robust Named Entity Recognizers From Noisy Data With Retrieval Augmentation [67.89838237013078]
  Named entity recognition (NER) models often struggle with noisy inputs. We propose a more realistic setting in which only noisy text and its NER labels are available. We employ a multi-view training framework that improves robust NER without retrieving text during inference.
  arXiv  Detail & Related papers  (2024-07-26T07:30:41Z)
• Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units [8.86336076082867]
  We propose a method for pretraining E2E KWS systems with untranscribed data. We show that finetuning such a model significantly outperforms a model trained from scratch.
  arXiv  Detail & Related papers  (2024-07-05T17:07:58Z)
• Augmenting text for spoken language understanding with Large Language Models [13.240782495441275]
  We show how to use transcript-semantic parse data (unpaired text) without corresponding speech. Experiments show that unpaired text from existing and new domains improves performance by 2% and 30% in absolute Exact Match (EM) respectively. We propose to prompt Large Language Models (LLMs) to generate unpaired text for existing and new domains.
  arXiv  Detail & Related papers  (2023-09-17T22:25:34Z)
• Text Injection for Capitalization and Turn-Taking Prediction in Speech Models [45.94388391693112]
  This study examines the use of text injection for auxiliary tasks, which are the non-ASR tasks often performed by an E2E model. We show results demonstrating that our text injection method boosts capitalization performance for long-tail data.
  arXiv  Detail & Related papers  (2023-08-14T18:28:04Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• BERM: Training the Balanced and Extractable Representation for Matching to Improve Generalization Ability of Dense Retrieval [54.66399120084227]
  We propose a novel method to improve the generalization of dense retrieval via capturing matching signal called BERM. Dense retrieval has shown promise in the first-stage retrieval process when trained on in-domain labeled datasets.
  arXiv  Detail & Related papers  (2023-05-18T15:43:09Z)
• RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models [12.37229805276939]
  We propose a novel pre-training method called Duplex Masked Auto-Encoder, a.k.a. DupMAE. It is designed to improve the quality semantic representation where all contextualized embeddings of the pretrained model can be leveraged.
  arXiv  Detail & Related papers  (2023-05-04T05:37:22Z)
• Leveraging Large Text Corpora for End-to-End Speech Summarization [58.673480990374635]
  End-to-end speech summarization (E2E SSum) is a technique to directly generate summary sentences from speech. We present two novel methods that leverage a large amount of external text summarization data for E2E SSum training.
  arXiv  Detail & Related papers  (2023-03-02T05:19:49Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• AutoSTR: Efficient Backbone Search for Scene Text Recognition [80.7290173000068]
  Scene text recognition (STR) is very challenging due to the diversity of text instances and the complexity of scenes. We propose automated STR (AutoSTR) to search data-dependent backbones to boost text recognition performance. Experiments demonstrate that, by searching data-dependent backbones, AutoSTR can outperform the state-of-the-art approaches on standard benchmarks.
  arXiv  Detail & Related papers  (2020-03-14T06:51:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.