Turning Whisper into Real-Time Transcription System

• URL: http://arxiv.org/abs/2307.14743v2
• Date: Thu, 21 Sep 2023 09:41:17 GMT
• Title: Turning Whisper into Real-Time Transcription System
• Abstract summary: Whisper-Streaming is an implementation of real-time speech transcription and translation of Whisper-like models. We show that Whisper-Streaming achieves high quality and 3.3 seconds latency on unsegmented long-form speech transcription test set.
• Score: 15.925481189003383
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Whisper is one of the recent state-of-the-art multilingual speech recognition and translation models, however, it is not designed for real time transcription. In this paper, we build on top of Whisper and create Whisper-Streaming, an implementation of real-time speech transcription and translation of Whisper-like models. Whisper-Streaming uses local agreement policy with self-adaptive latency to enable streaming transcription. We show that Whisper-Streaming achieves high quality and 3.3 seconds latency on unsegmented long-form speech transcription test set, and we demonstrate its robustness and practical usability as a component in live transcription service at a multilingual conference.

Related papers

• Context-Aware Interleaved Batching for WhisperX [3.7355752644425917]
  Our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous context across batched audio segments.<n>As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER)
  arXiv  Detail & Related papers  (2026-08-31T17:59:46Z)
• Voxtral Realtime [134.66962524291424]
  Voxtral Realtime is a streaming automatic speech recognition model.<n>It matches offline transcription quality at sub-second latency.<n>We release the model weights under the Apache 2.0 license.
  arXiv  Detail & Related papers  (2026-02-11T19:17:10Z)
• Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing [15.134076873312809]
  Cross-lingual dubbing system translates speech from one language to another while preserving key characteristics such as duration, speaker identity, and speaking speed.<n>We propose a discrete diffusion-based speech-to-unit translation model with explicit duration control, enabling time-aligned translation.<n>We then synthesize speech based on the predicted units and source identity with a conditional flow matching model.
  arXiv  Detail & Related papers  (2025-05-27T08:43:28Z)
• High-Fidelity Simultaneous Speech-To-Speech Translation [75.69884829562591]
  We introduce Hibiki, a decoder-only model for simultaneous speech translation. Hibiki leverages a multistream language model to synchronously process source and target speech, and jointly produces text and audio tokens to perform speech-to-text and speech-to-speech translation.
  arXiv  Detail & Related papers  (2025-02-05T17:18:55Z)
• Enhancing Whisper's Accuracy and Speed for Indian Languages through Prompt-Tuning and Tokenization [2.403252956256118]
  This paper explores two novel approaches to enhance Whisper's multilingual speech recognition performance in Indian languages. First, we propose prompt-tuning with language family information, which enhances Whisper's accuracy in linguistically similar languages. Second, we introduce a novel tokenizer that reduces the number of generated tokens, thereby accelerating Whisper's inference speed.
  arXiv  Detail & Related papers  (2024-12-27T18:32:24Z)
• CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions [0.5120567378386615]
  We fine-tune the model to produce more verbatim speech transcriptions. We employ several techniques to increase robustness against multiple speakers and background noise.
  arXiv  Detail & Related papers  (2024-08-29T14:52:42Z)
• Cross-Lingual Transfer Learning for Speech Translation [7.802021866251242]
  This paper examines how to expand the speech translation capability of speech foundation models with restricted data. Whisper, a speech foundation model with strong performance on speech recognition and English translation, is used as the example model. Using speech-to-speech retrieval to analyse the audio representations generated by the encoder, we show that utterances from different languages are mapped to a shared semantic space.
  arXiv  Detail & Related papers  (2024-07-01T09:51:48Z)
• Whispy: Adapting STT Whisper Models to Real-Time Environments [43.000853406349165]
  We introduce Whispy, a system to bring live capabilities to Whisper pretrained models. Whispy is able to consume live audio streams and generate high level, coherent voice transcriptions. We evaluate the performance of our system on a large repository of publicly available speech datasets.
  arXiv  Detail & Related papers  (2024-05-06T13:55:39Z)
• Seamless: Multilingual Expressive and Streaming Speech Translation [71.12826355107889]
  We introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model- SeamlessM4T v2. We bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time.
  arXiv  Detail & Related papers  (2023-12-08T17:18:42Z)
• Leveraging Timestamp Information for Serialized Joint Streaming Recognition and Translation [51.399695200838586]
  We propose a streaming Transformer-Transducer (T-T) model able to jointly produce many-to-one and one-to-many transcription and translation using a single decoder. Experiments on it,es,de->en prove the effectiveness of our approach, enabling the generation of one-to-many joint outputs with a single decoder for the first time.
  arXiv  Detail & Related papers  (2023-10-23T11:00:27Z)
• DiariST: Streaming Speech Translation with Speaker Diarization [53.595990270899414]
  We propose DiariST, the first streaming ST and SD solution. It is built upon a neural transducer-based streaming ST system and integrates token-level serialized output training and t-vector. Our system achieves a strong ST and SD capability compared to offline systems based on Whisper, while performing streaming inference for overlapping speech.
  arXiv  Detail & Related papers  (2023-09-14T19:33:27Z)
• Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing [71.02335065794384]
  We propose a model that directly optimize both the translation as well as the speech duration of the generated translations. We show that this system generates speech that better matches the timing of the original speech, compared to prior work, while simplifying the system architecture.
  arXiv  Detail & Related papers  (2023-02-25T04:23:25Z)
• Enabling Interactive Transcription in an Indigenous Community [23.53585157238112]
  We propose a novel transcription workflow which combines spoken term detection and human-in-the-loop. We show that in the early stages of transcription, when the available data is insufficient to train a robust ASR system, it is possible to take advantage of the transcription of a small number of isolated words.
  arXiv  Detail & Related papers  (2020-11-12T04:41:35Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)
• Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training [40.71155396456831]
  Simultaneous speech-to-speech translation is widely useful but extremely challenging. It needs to generate target-language speech concurrently with the source-language speech, with only a few seconds delay. Current approaches accumulate latencies progressively when the speaker talks faster, and introduce unnatural pauses when the speaker talks slower. We propose Self-Adaptive Translation (SAT) which flexibly adjusts the length of translations to accommodate different source speech rates.
  arXiv  Detail & Related papers  (2020-10-20T06:02:15Z)
• Textual Supervision for Visually Grounded Spoken Language Understanding [51.93744335044475]
  Visually-grounded models of spoken language understanding extract semantic information directly from speech. This is useful for low-resource languages, where transcriptions can be expensive or impossible to obtain. Recent work showed that these models can be improved if transcriptions are available at training time.
  arXiv  Detail & Related papers  (2020-10-06T15:16:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.