Whispy: Adapting STT Whisper Models to Real-Time Environments

• URL: http://arxiv.org/abs/2405.03484v1
• Date: Mon, 6 May 2024 13:55:39 GMT
• Title: Whispy: Adapting STT Whisper Models to Real-Time Environments
• Abstract summary: We introduce Whispy, a system to bring live capabilities to Whisper pretrained models. Whispy is able to consume live audio streams and generate high level, coherent voice transcriptions. We evaluate the performance of our system on a large repository of publicly available speech datasets.
• Score: 43.000853406349165
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large general-purpose transformer models have recently become the mainstay in the realm of speech analysis. In particular, Whisper achieves state-of-the-art results in relevant tasks such as speech recognition, translation, language identification, and voice activity detection. However, Whisper models are not designed to be used in real-time conditions, and this limitation makes them unsuitable for a vast plethora of practical applications. In this paper, we introduce Whispy, a system intended to bring live capabilities to the Whisper pretrained models. As a result of a number of architectural optimisations, Whispy is able to consume live audio streams and generate high level, coherent voice transcriptions, while still maintaining a low computational cost. We evaluate the performance of our system on a large repository of publicly available speech datasets, investigating how the transcription mechanism introduced by Whispy impacts on the Whisper output. Experimental results show how Whispy excels in robustness, promptness, and accuracy.

Related papers

• Pisets: A Robust Speech Recognition System for Lectures and Interviews [2.0524609401792397]
  This work presents a speech-to-text system "Pisets" for scientists and journalists.<n>The architecture comprises primary recognition using Wav2Vec2, false positive filtering via the Audio Spectrogram Transformer (AST), and final speech recognition through Whisper.<n>The proposed approaches ensure robust transcribing of long audio data across various acoustic conditions compared to WhisperX and the usual Whisper model.
  arXiv  Detail & Related papers  (2026-01-26T12:14:51Z)
• Defense Against Synthetic Speech: Real-Time Detection of RVC Voice Conversion Attacks [0.0]
  Generative audio technologies now enable highly realistic voice cloning and real-time voice conversion.<n>This study investigates real-time detection of AI-generated speech produced using Retrieval-based Voice Conversion.
  arXiv  Detail & Related papers  (2025-12-31T02:06:42Z)
• Measuring the Robustness of Audio Deepfake Detectors [59.09338266364506]
  This work systematically evaluates the robustness of 10 audio deepfake detection models against 16 common corruptions. Using both traditional deep learning models and state-of-the-art foundation models, we make four unique observations.
  arXiv  Detail & Related papers  (2025-03-21T23:21:17Z)
• Whisper Finetuning on Nepali Language [0.0]
  This research focuses on making an exhaustive and generalized dataset followed by fine-tuning OpenAI's Whisper models to improve transcription accuracy for the Nepali language. We leverage publicly available ASR datasets and self-recorded custom datasets with a diverse range of accents, dialects, and speaking styles further enriched through augmentation. Our approach outperforms Whisper's baseline models trained on Fleur's dataset, achieving WER reductions of up to 36.2% on the small and 23.8% on medium models.
  arXiv  Detail & Related papers  (2024-11-19T15:55:56Z)
• Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning [55.2480439325792]
  Large audio-language models (LALMs) have shown impressive capabilities in understanding and reasoning about audio and speech information. These models still face challenges, including hallucinating non-existent sound events, misidentifying the order of sound events, and incorrectly attributing sound sources.
  arXiv  Detail & Related papers  (2024-10-21T15:55:27Z)
• FINALLY: fast and universal speech enhancement with studio-like quality [7.207284147264852]
  We address the challenge of speech enhancement in real-world recordings, which often contain various forms of distortion. We study various feature extractors for perceptual loss to facilitate the stability of adversarial training. We integrate WavLM-based perceptual loss into MS-STFT adversarial training pipeline, creating an effective and stable training procedure for the speech enhancement model.
  arXiv  Detail & Related papers  (2024-10-08T11:16:03Z)
• Incorporating Talker Identity Aids With Improving Speech Recognition in Adversarial Environments [0.2916558661202724]
  We develop a transformer-based model that jointly performs speech recognition and speaker identification. We show that the joint model performs comparably to Whisper under clean conditions. Our results suggest that integrating voice representations with speech recognition can lead to more robust models under adversarial conditions.
  arXiv  Detail & Related papers  (2024-10-07T18:39:59Z)
• TIGER: Time-frequency Interleaved Gain Extraction and Reconstruction for Efficient Speech Separation [19.126525226518975]
  We propose a speech separation model with significantly reduced parameters and computational costs. TIGER leverages prior knowledge to divide frequency bands and compresses frequency information. We show that TIGER achieves performance surpassing state-of-the-art (SOTA) model TF-GridNet.
  arXiv  Detail & Related papers  (2024-10-02T12:21:06Z)
• SpeechPrompt: Prompting Speech Language Models for Speech Processing Tasks [94.10497337235083]
  We are first to explore the potential of prompting speech LMs in the domain of speech processing. We reformulate speech processing tasks into speech-to-unit generation tasks. We show that the prompting method can achieve competitive performance compared to the strong fine-tuning method.
  arXiv  Detail & Related papers  (2024-08-23T13:00:10Z)
• A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement [16.900731393703648]
  Self-supervised learned models have been found to be very effective for certain speech tasks. In this paper, we investigate the uses of SSL representations for single-channel speech enhancement in challenging conditions.
  arXiv  Detail & Related papers  (2024-03-03T02:05:17Z)
• Pheme: Efficient and Conversational Speech Generation [52.34331755341856]
  We introduce the Pheme model series that offers compact yet high-performing conversational TTS models. It can be trained efficiently on smaller-scale conversational data, cutting data demands by more than 10x but still matching the quality of the autoregressive TTS models.
  arXiv  Detail & Related papers  (2024-01-05T14:47:20Z)
• Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study [68.88536866933038]
  Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length.
  arXiv  Detail & Related papers  (2023-09-27T17:21:13Z)
• Speech Enhancement for Wake-Up-Word detection in Voice Assistants [60.103753056973815]
  Keywords spotting and in particular Wake-Up-Word (WUW) detection is a very important task for voice assistants. This paper proposes a Speech Enhancement model adapted to the task of WUW detection. It aims at increasing the recognition rate and reducing the false alarms in the presence of these types of noises.
  arXiv  Detail & Related papers  (2021-01-29T18:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.