ASTRA: Aligning Speech and Text Representations for Asr without Sampling

• URL: http://arxiv.org/abs/2406.06664v2
• Date: Thu, 13 Jun 2024 15:39:03 GMT
• Title: ASTRA: Aligning Speech and Text Representations for Asr without Sampling
• Authors: Neeraj Gaur, Rohan Agrawal, Gary Wang, Parisa Haghani, Andrew Rosenberg, Bhuvana Ramabhadran,
• Abstract summary: ASTRA is a novel method for improving Automatic Speech Recognition (ASR) through text injection. Unlike prevailing techniques, ASTRA eliminates the need for sampling to match sequence lengths between speech and text modalities.
• Score: 20.925353958092874
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper introduces ASTRA, a novel method for improving Automatic Speech Recognition (ASR) through text injection.Unlike prevailing techniques, ASTRA eliminates the need for sampling to match sequence lengths between speech and text modalities. Instead, it leverages the inherent alignments learned within CTC/RNNT models. This approach offers the following two advantages, namely, avoiding potential misalignment between speech and text features that could arise from upsampling and eliminating the need for models to accurately predict duration of sub-word tokens. This novel formulation of modality (length) matching as a weighted RNNT objective matches the performance of the state-of-the-art duration-based methods on the FLEURS benchmark, while opening up other avenues of research in speech processing.

Related papers

• BEST-STD: Bidirectional Mamba-Enhanced Speech Tokenization for Spoken Term Detection [8.303512060791736]
  Spoken term detection is often hindered by reliance on frame-level features and the computationally intensive DTW-based template matching. We propose a novel approach that encodes speech into discrete, speaker-agnostic semantic tokens. This facilitates fast retrieval using text-based search algorithms and effectively handles out-of-vocabulary terms.
  arXiv  Detail & Related papers  (2024-11-21T13:05:18Z)
• SegINR: Segment-wise Implicit Neural Representation for Sequence Alignment in Neural Text-to-Speech [13.82115484420239]
  We present SegINR, a novel approach to neural Text-to-Speech (TTS) It addresses sequence alignment without relying on an auxiliary duration predictor and complex autoregressive (AR) or non-autoregressive (NAR) frame-level sequence modeling. Our experiments in zero-shot adaptive TTS scenarios demonstrate that SegINR outperforms conventional methods in speech quality with computational efficiency.
  arXiv  Detail & Related papers  (2024-10-07T02:04:58Z)
• Out of Length Text Recognition with Sub-String Matching [54.63761108308825]
  In this paper, we term this task Out of Length (OOL) text recognition. We propose a novel method called OOL Text Recognition with sub-String Matching (SMTR) SMTR comprises two cross-attention-based modules: one encodes a sub-string containing multiple characters into next and previous queries, and the other employs the queries to attend to the image features.
  arXiv  Detail & Related papers  (2024-07-17T05:02:17Z)
• Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representation [67.98338382984556]
  Mapping two modalities, speech and text, into a shared representation space, is a research topic of using text-only data to improve end-to-end automatic speech recognition (ASR) performance in new domains. In this paper, we proposed novel representations match strategy through down-sampling acoustic representation to align with text modality. Our ASR model can learn unified representations from both modalities better, allowing for domain adaptation using text-only data of the target domain.
  arXiv  Detail & Related papers  (2023-09-04T08:52:59Z)
• AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment [67.10208647482109]
  The speech-to-singing (STS) voice conversion task aims to generate singing samples corresponding to speech recordings. This paper proposes AlignSTS, an STS model based on explicit cross-modal alignment. Experiments show that AlignSTS achieves superior performance in terms of both objective and subjective metrics.
  arXiv  Detail & Related papers  (2023-05-08T06:02:10Z)
• Code-Switching Text Generation and Injection in Mandarin-English ASR [57.57570417273262]
  We investigate text generation and injection for improving the performance of an industry commonly-used streaming model, Transformer-Transducer (T-T) We first propose a strategy to generate code-switching text data and then investigate injecting generated text into T-T model explicitly by Text-To-Speech (TTS) conversion or implicitly by tying speech and text latent spaces. Experimental results on the T-T model trained with a dataset containing 1,800 hours of real Mandarin-English code-switched speech show that our approaches to inject generated code-switching text significantly boost the performance of T-T models.
  arXiv  Detail & Related papers  (2023-03-20T09:13:27Z)
• Discrete Cross-Modal Alignment Enables Zero-Shot Speech Translation [71.35243644890537]
  End-to-end Speech Translation (ST) aims at translating the source language speech into target language text without generating the intermediate transcriptions. Existing zero-shot methods fail to align the two modalities of speech and text into a shared semantic space. We propose a novel Discrete Cross-Modal Alignment (DCMA) method that employs a shared discrete vocabulary space to accommodate and match both modalities of speech and text.
  arXiv  Detail & Related papers  (2022-10-18T03:06:47Z)
• Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformers [49.403414751667135]
  This paper proposes a novel label-synchronous speech-to-text alignment technique for automatic speech recognition (ASR) The proposed method re-defines the speech-to-text alignment as a label-synchronous text mapping problem. Experiments using the corpus of spontaneous Japanese (CSJ) demonstrate that the proposed method provides an accurate utterance-wise alignment.
  arXiv  Detail & Related papers  (2021-04-21T03:05:12Z)
• End to End ASR System with Automatic Punctuation Insertion [0.0]
  We propose a method to generate punctuated transcript for the TEDLIUM dataset using transcripts available from ted.com. We also propose an end-to-end ASR system that outputs words and punctuations concurrently from speech signals.
  arXiv  Detail & Related papers  (2020-12-03T15:46:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.