CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword Spotting

• URL: http://arxiv.org/abs/2406.07923v1
• Date: Wed, 12 Jun 2024 06:44:40 GMT
• Title: CTC-aligned Audio-Text Embedding for Streaming Open-vocabulary Keyword Spotting
• Authors: Sichen Jin, Youngmoon Jung, Seungjin Lee, Jaeyoung Roh, Changwoo Han, Hoonyoung Cho,
• Abstract summary: This paper introduces a novel approach for streaming openvocabulary keyword spotting (KWS) with text-based keyword enrollment. For every input frame, the proposed method finds the optimal alignment ending at the frame using connectionist temporal classification (CTC) We then aggregates the frame-level acoustic embedding (AE) to obtain higher-level (i.e., character, word, or phrase) AE that aligns with the text embedding (TE) of the target keyword text.
• Score: 6.856101216726412
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a novel approach for streaming openvocabulary keyword spotting (KWS) with text-based keyword enrollment. For every input frame, the proposed method finds the optimal alignment ending at the frame using connectionist temporal classification (CTC) and aggregates the frame-level acoustic embedding (AE) to obtain higher-level (i.e., character, word, or phrase) AE that aligns with the text embedding (TE) of the target keyword text. After that, we calculate the similarity of the aggregated AE and the TE. To the best of our knowledge, this is the first attempt to dynamically align the audio and the keyword text on-the-fly to attain the joint audio-text embedding for KWS. Despite operating in a streaming fashion, our approach achieves competitive performance on the LibriPhrase dataset compared to the non-streaming methods with a mere 155K model parameters and a decoding algorithm with time complexity O(U), where U is the length of the target keyword at inference time.

Related papers

• VQ-CTAP: Cross-Modal Fine-Grained Sequence Representation Learning for Speech Processing [81.32613443072441]
  For tasks such as text-to-speech (TTS), voice conversion (VC), and automatic speech recognition (ASR), a cross-modal fine-grained (frame-level) sequence representation is desired. We propose a method called Quantized Contrastive Token-Acoustic Pre-training (VQ-CTAP), which uses the cross-modal sequence transcoder to bring text and speech into a joint space.
  arXiv  Detail & Related papers  (2024-08-11T12:24:23Z)
• VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment [101.2489492032816]
  VALL-E R is a robust and efficient zero-shot Text-to-Speech system. This research has the potential to be applied to meaningful projects, including the creation of speech for those affected by aphasia.
  arXiv  Detail & Related papers  (2024-06-12T04:09:44Z)
• Relational Proxy Loss for Audio-Text based Keyword Spotting [8.932603220365793]
  This study aims to improve existing methods by leveraging the structural acoustic embeddings and within text embeddings. By incorporating RPL, we demonstrated improved performance on the Wall Street Journal (WSJ) corpus.
  arXiv  Detail & Related papers  (2024-06-08T01:21:17Z)
• Matching Latent Encoding for Audio-Text based Keyword Spotting [9.599402723927733]
  We propose an audio-text-based end-to-end model architecture for flexible keyword spotting (KWS) Our architecture uses a novel dynamic programming-based algorithm, Dynamic Sequence Partitioning (DSP), to optimally partition the audio sequence into the same length as the word-based text sequence. Experimental results show that our DSP is more effective than other partitioning schemes.
  arXiv  Detail & Related papers  (2023-06-08T14:44:23Z)
• Iterative pseudo-forced alignment by acoustic CTC loss for self-supervised ASR domain adaptation [80.12316877964558]
  High-quality data labeling from specific domains is costly and human time-consuming. We propose a self-supervised domain adaptation method, based upon an iterative pseudo-forced alignment algorithm.
  arXiv  Detail & Related papers  (2022-10-27T07:23:08Z)
• Learning Audio-Text Agreement for Open-vocabulary Keyword Spotting [23.627625026135505]
  We propose a novel end-to-end user-defined keyword spotting method. Our method compares input queries with an enrolled text keyword sequence. We introduce the LibriPhrase dataset for efficiently training keyword spotting models.
  arXiv  Detail & Related papers  (2022-06-30T16:40:31Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• SCATTER: Selective Context Attentional Scene Text Recognizer [16.311256552979835]
  Scene Text Recognition (STR) is the task of recognizing text against complex image backgrounds. Current state-of-the-art (SOTA) methods still struggle to recognize text written in arbitrary shapes. We introduce a novel architecture for STR, named Selective Context ATtentional Text Recognizer (SCATTER)
  arXiv  Detail & Related papers  (2020-03-25T09:20:28Z)
• End-to-End Automatic Speech Recognition Integrated With CTC-Based Voice Activity Detection [48.80449801938696]
  This paper integrates a voice activity detection function with end-to-end automatic speech recognition. We focus on connectionist temporal classification ( CTC) and its extension ofsynchronous/attention. We use the labels as a cue for detecting speech segments with simple thresholding.
  arXiv  Detail & Related papers  (2020-02-03T03:36:34Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.