Context-Aware Interleaved Batching for WhisperX

• URL: http://arxiv.org/abs/2608.31170v1
• Date: Mon, 31 Aug 2026 17:59:46 GMT
• Title: Context-Aware Interleaved Batching for WhisperX
• Abstract summary: Our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous context across batched audio segments.<n>As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER)
• Score: 3.7355752644425917
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While WhisperX accelerates speech transcription via intra-audio batching, it isolates audio segments, losing the historical context needed for coherent punctuation and terminology transcription. Conversely, standard Whisper retains context sequentially but suffers from slow inference and hallucination loops. To achieve the best of both worlds, we propose Context-Aware Interleaved Batching. By using VAD-derived segment boundaries, our algorithm stabilizes Whisper's text conditioning, allowing us to safely maintain continuous historical context across batched audio segments. As demonstrated on long-form audio benchmarks, this approach reduces Word Error Rate (WER) and improves proper noun transcription, all while maintaining high-throughput inference speeds.

Related papers

• X2Streaming-TTS: Causal Token-Level Text-to-Speech from Streaming Text with Speech-State Inheritance [4.713402367550747]
  Streaming text-to-speech is essential for low-latency spoken dialogue systems, but many systems wait for sentence-level text and are therefore only pseudo-streaming.<n>We present X2Streaming-TTS, a causal TTS framework that consumes asynchronously arriving text tokens and emits speech without accessing future input.
  arXiv  Detail & Related papers  (2026-08-19T08:05:45Z)
• CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing [76.74048814837336]
  Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video.<n>Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level.<n>We propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT)
  arXiv  Detail & Related papers  (2026-04-14T05:03:57Z)
• Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR [14.082117520000873]
  We study whether multimodal context from prior turns improves LLM-based ASR.<n>We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities.<n>We propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly.
  arXiv  Detail & Related papers  (2026-03-27T10:09:30Z)
• RFM-Editing: Rectified Flow Matching for Text-guided Audio Editing [21.479883699581308]
  We propose a novel end-to-end efficient rectified flow matching-based diffusion framework for audio editing.<n> Experiments show that our model achieves faithful semantic alignment without requiring auxiliary captions or masks.
  arXiv  Detail & Related papers  (2025-09-17T14:13:40Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• Character-aware audio-visual subtitling in context [58.95580154761008]
  This paper presents an improved framework for character-aware audio-visual subtitling in TV shows. Our approach integrates speech recognition, speaker diarisation, and character recognition, utilising both audio and visual cues. We validate the method on a dataset with 12 TV shows, demonstrating superior performance in speaker diarisation and character recognition accuracy compared to existing approaches.
  arXiv  Detail & Related papers  (2024-10-14T20:27:34Z)
• CrisperWhisper: Accurate Timestamps on Verbatim Speech Transcriptions [0.5120567378386615]
  We fine-tune the model to produce more verbatim speech transcriptions. We employ several techniques to increase robustness against multiple speakers and background noise.
  arXiv  Detail & Related papers  (2024-08-29T14:52:42Z)
• Turning Whisper into Real-Time Transcription System [15.925481189003383]
  Whisper-Streaming is an implementation of real-time speech transcription and translation of Whisper-like models. We show that Whisper-Streaming achieves high quality and 3.3 seconds latency on unsegmented long-form speech transcription test set.
  arXiv  Detail & Related papers  (2023-07-27T10:00:05Z)
• Efficient Audio Captioning Transformer with Patchout and Text Guidance [74.59739661383726]
  We propose a full Transformer architecture that utilizes Patchout as proposed in [1], significantly reducing the computational complexity and avoiding overfitting. The caption generation is partly conditioned on textual AudioSet tags extracted by a pre-trained classification model. Our proposed method received the Judges Award at the Task6A of DCASE Challenge 2022.
  arXiv  Detail & Related papers  (2023-04-06T07:58:27Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• Zero-Shot Text-to-Speech for Text-Based Insertion in Audio Narration [62.75234183218897]
  We propose a one-stage context-aware framework to generate natural and coherent target speech without any training data of the speaker. We generate the mel-spectrogram of the edited speech with a transformer-based decoder. It outperforms a recent zero-shot TTS engine by a large margin.
  arXiv  Detail & Related papers  (2021-09-12T04:17:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.