Voxtral Realtime

• URL: http://arxiv.org/abs/2602.11298v1
• Date: Wed, 11 Feb 2026 19:17:10 GMT
• Title: Voxtral Realtime
• Abstract summary: Voxtral Realtime is a streaming automatic speech recognition model.<n>It matches offline transcription quality at sub-second latency.<n>We release the model weights under the Apache 2.0 license.
• Score: 134.66962524291424
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Voxtral Realtime, a natively streaming automatic speech recognition model that matches offline transcription quality at sub-second latency. Unlike approaches that adapt offline models through chunking or sliding windows, Voxtral Realtime is trained end-to-end for streaming, with explicit alignment between audio and text streams. Our architecture builds on the Delayed Streams Modeling framework, introducing a new causal audio encoder and Ada RMS-Norm for improved delay conditioning. We scale pretraining to a large-scale dataset spanning 13 languages. At a delay of 480ms, Voxtral Realtime achieves performance on par with Whisper, the most widely deployed offline transcription system. We release the model weights under the Apache 2.0 license.

Related papers

• Typhoon ASR Streaming: Steerable Low-Latency Thai Speech Recognition with Real-Time Shallow Fusion [2.6499147105354077]
  Open Thai automatic speech recognition (ASR) is dominated by offline, Whisper-based models that read the whole utterance before transcribing.<n>We present a deployable system for streaming Thai ASR that lets a user steer its vocabulary at decode time, without retraining.
  arXiv  Detail & Related papers  (2026-09-14T03:53:52Z)
• Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models [68.51256516936338]
  Wan-Streamer is an end-to-end native interactive foundation model for low-streaming interaction.<n>It seamlessly models audio and video as both input and output within single Transformer sequence.<n>It achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency.
  arXiv  Detail & Related papers  (2026-06-23T18:01:03Z)
• Audio Interaction Model [102.4354125819644]
  Today's Large Audio Language Models (LALMs) are offline, and streaming audio models each handle only a single task such as streaming ASR or voice chatting.<n>It is time to unify them into one online LALM: a model that, through an always-on perceive-decide-respond loop, listens to sound, environment, and instructions in real time and reacts on the fly.<n>We formalize this regime as the Audio Interaction Model, and realize it with Audio-Interaction, a unified streaming model that retains offline task execution while adding online general audio instruction following.
  arXiv  Detail & Related papers  (2026-06-03T17:26:11Z)
• Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models [56.91801348360746]
  Large audio-language models (LALMs) generalize across speech, sound, and music.<n> unified decoders can exhibit a emphtemporal smoothing bias<n>We propose emphTemporal Contrastive Decoding (TCD), a training-free decoding method for unified LALMs.
  arXiv  Detail & Related papers  (2026-04-16T02:30:41Z)
• Real-Time Streamable Generative Speech Restoration with Flow Matching [35.33575179870606]
  Stream$.$FM is a frame-causal flow-based generative model with an algorithmic latency of 32 milliseconds (ms) and a total latency of 48 ms.<n>We show that high-quality streaming generative speech processing can be realized on consumer GPUs available today.
  arXiv  Detail & Related papers  (2025-12-22T14:41:17Z)
• VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency [17.067283475630095]
  We present VoXtream, a fully autoregressive, zero-shot streaming text-to-speech (TTS) system for real-time use that begins speaking from the first word.<n>VoXtream directly maps incoming phonemes to audio tokens using a monotonic alignment scheme and a dynamic look-ahead that does not delay onset.<n>Built around an incremental phoneme transformer, a temporal transformer predicting semantic and duration tokens, and a depth transformer producing acoustic tokens, VoXtream achieves, to our knowledge, the lowest initial delay among publicly available streaming TTS: 102 ms on
  arXiv  Detail & Related papers  (2025-09-19T13:26:46Z)
• CarelessWhisper: Turning Whisper into a Causal Streaming Model [31.38962687054824]
  We present an analysis explaining why it is not straightforward to convert an encoder-decoder transformer to a low-latency streaming model.<n>Our proposed method modifies the existing (non-causal) encoder to a causal encoder by fine-tuning both the encoder and decoder.<n> Experiments on low-latency chunk sizes (less than 300 msec) show that our fine-tuned model outperforms existing non-fine-tuned streaming approaches.
  arXiv  Detail & Related papers  (2025-08-17T09:32:40Z)
• Streaming Non-Autoregressive Model for Accent Conversion and Pronunciation Improvement [52.89324095217975]
  We propose a first streaming accent conversion model that transforms non-native speech into a native-like accent.<n>Our approach enables stream processing by modifying a previous AC architecture with an Emformer encoder and an optimized inference mechanism.
  arXiv  Detail & Related papers  (2025-06-19T20:05:29Z)
• StreamMel: Real-Time Zero-shot Text-to-Speech via Interleaved Continuous Autoregressive Modeling [50.537794606598254]
  StreamMel is a pioneering single-stage streaming TTS framework that models continuous mel-spectrograms.<n>It enables low-latency, autoregressive synthesis while preserving high speaker similarity and naturalness.<n>It even achieves performance comparable to offline systems while supporting efficient real-time generation.
  arXiv  Detail & Related papers  (2025-06-14T16:53:39Z)
• Streaming Audio-Visual Speech Recognition with Alignment Regularization [69.30185151873707]
  We propose a streaming AV-ASR system based on a hybrid connectionist temporal classification ( CTC)/attention neural network architecture. The proposed AV-ASR model achieves WERs of 2.0% and 2.6% on the Lip Reading Sentences 3 dataset in an offline and online setup.
  arXiv  Detail & Related papers  (2022-11-03T20:20:47Z)
• End to End Lip Synchronization with a Temporal AutoEncoder [95.94432031144716]
  We study the problem of syncing the lip movement in a video with the audio stream. Our solution finds an optimal alignment using a dual-domain recurrent neural network. As an application, we demonstrate our ability to robustly align text-to-speech generated audio with an existing video stream.
  arXiv  Detail & Related papers  (2022-03-30T12:00:18Z)
• SoundStream: An End-to-End Neural Audio Codec [78.94923131038682]
  We present SoundStream, a novel neural audio system that can efficiently compress speech, music and general audio. SoundStream relies on a fully convolutional encoder/decoder network and a residual vector quantizer, which are trained jointly end-to-end. We are able to perform joint compression and enhancement either at the encoder or at the decoder side with no additional latency.
  arXiv  Detail & Related papers  (2021-07-07T15:45:42Z)
• Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition [16.082949461807335]
  We present a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. We show that we can run this model in a Y-model architecture with the top layers running in parallel in low latency and high latency modes. This allows us to have streaming speech recognition results with limited latency and delayed speech recognition results with large improvements in accuracy.
  arXiv  Detail & Related papers  (2020-10-07T05:58:28Z)
• Streaming automatic speech recognition with the transformer model [59.58318952000571]
  We propose a transformer based end-to-end ASR system for streaming ASR. We apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech.
  arXiv  Detail & Related papers  (2020-01-08T18:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.