Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization

• URL: http://arxiv.org/abs/2604.19079v1
• Date: Tue, 21 Apr 2026 04:43:03 GMT
• Title: Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization
• Abstract summary: We present a Unified ASR framework for Transducer (RNNT) training that supports both offline and streaming decoding within a single model.<n>Experiments show that the proposed approach improves streaming accuracy at low latency while preserving offline performance and scaling to larger model sizes and training datasets.
• Score: 35.546900956309294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unification of automatic speech recognition (ASR) systems reduces development and maintenance costs, but training a single model to perform well in both offline and low-latency streaming settings remains challenging. We present a Unified ASR framework for Transducer (RNNT) training that supports both offline and streaming decoding within a single model, using chunk-limited attention with right context and dynamic chunked convolutions. To further close the gap between offline and streaming performance, we introduce an efficient Triton implementation of mode-consistency regularization for RNNT (MCR-RNNT), which encourages agreement across training modes. Experiments show that the proposed approach improves streaming accuracy at low latency while preserving offline performance and scaling to larger model sizes and training datasets. The proposed Unified ASR framework and the English model checkpoint are open-sourced.

Related papers

• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• SATB-VR: Training Few-Step Video Restoration Diffusion Model using SNR-Aware Trajectory Blending [55.764018145256216]
  We present SATB-VR, a few-step paradigm that jump-starts the denoising process via an auxiliary predictor.<n>We show that SATB-VR performs favorably against existing approaches on synthetic, real-world, and AIGC benchmarks.
  arXiv  Detail & Related papers  (2026-06-27T01:32:16Z)
• FLARE: Diffusion for Hybrid Language Model [72.60770374799634]
  FLARE is a systematic conversion framework for hybrid-attention large language models.<n>It combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising.<n>Our results suggest practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives.
  arXiv  Detail & Related papers  (2026-06-01T06:58:15Z)
• Accelerating Structured Chain-of-Thought in Autonomous Vehicles [55.78214218065611]
  Chain-of-Thought (CoT) reasoning enhances the decision-making capabilities of vision-language-action models in autonomous driving.<n>FastDriveCoT is a novel parallel decoding method that accelerates template-structured CoT.
  arXiv  Detail & Related papers  (2026-02-02T22:14:26Z)
• Causal Autoregressive Diffusion Language Model [70.7353007255797]
  CARD reformulates the diffusion process within a strictly causal attention mask, enabling dense, per-token supervision in a single forward pass.<n>Our results demonstrate that CARD achieves ARM-level data efficiency while unlocking the latency benefits of parallel generation.
  arXiv  Detail & Related papers  (2026-01-29T17:38:29Z)
• Real-Time Inference for Distributed Multimodal Systems under Communication Delay Uncertainty [37.15356899831919]
  Connected cyber-physical systems perform inference based on real-time inputs from multiple data streams.<n>We propose a novel neuro-inspired non-blocking inference paradigm that employs adaptive temporal windows of integration.<n>Our framework achieves robust real-time inference with finer-grained control over the accuracy-latency tradeoff.
  arXiv  Detail & Related papers  (2025-11-20T10:48:54Z)
• BADiff: Bandwidth Adaptive Diffusion Model [55.10134744772338]
  Traditional diffusion models produce high-fidelity images by performing a fixed number of denoising steps, regardless of downstream transmission limitations.<n>In practical cloud-to-device scenarios, limited bandwidth often necessitates heavy compression, leading to loss of fine textures and wasted computation.<n>We introduce a joint end-to-end training strategy where the diffusion model is conditioned on a target quality level derived from the available bandwidth.
  arXiv  Detail & Related papers  (2025-10-24T11:50:03Z)
• Unifying Streaming and Non-streaming Zipformer-based ASR [14.226219579716629]
  We present a unified framework that trains a single end-to-end ASR model for both streaming and non-streaming applications.<n>We propose to use dynamic right-context through the chunked attention masking in the training of zipformer-based ASR models.<n>We analyze the effect of varying the number of right-context frames on accuracy and latency of the streaming ASR models.
  arXiv  Detail & Related papers  (2025-06-17T11:52:41Z)
• HENT-SRT: Hierarchical Efficient Neural Transducer with Self-Distillation for Joint Speech Recognition and Translation [19.997594859651233]
  HENT-SRT is a novel framework that factorizes ASR and translation tasks to better handle reordering.<n>We improve computational efficiency by incorporating best practices from ASR transducers.<n>Our approach is evaluated on three conversational datasets Arabic, Spanish, and Mandarin.
  arXiv  Detail & Related papers  (2025-06-02T18:37:50Z)
• DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR [20.42366884075422]
  We propose the integration of a novel dynamic contextual carry-over mechanism in a state-of-the-art unified ASR system. Our proposed dynamic context Conformer (DCTX-Conformer) utilizes a non-overlapping contextual carry-over mechanism. We outperform the SOTA by a relative 25.0% word error rate, with a negligible latency impact due to the additional context embeddings.
  arXiv  Detail & Related papers  (2023-06-13T23:42:53Z)
• Streaming Align-Refine for Non-autoregressive Deliberation [42.748839817396046]
  We propose a streaming non-autoregressive (non-AR) decoding algorithm to deliberate the hypothesis alignment of a streaming RNN-T model. Our algorithm facilitates a simple greedy decoding procedure, and at the same time is capable of producing the decoding result at each frame with limited right context. Experiments on voice search datasets and Librispeech show that with reasonable right context, our streaming model performs as well as the offline counterpart.
  arXiv  Detail & Related papers  (2022-04-15T17:24:39Z)
• Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition [58.69803243323346]
  Attention-based end-to-end automatic speech recognition (ASR) systems have recently demonstrated state-of-the-art results for numerous tasks. However, the application of self-attention and attention-based encoder-decoder models remains challenging for streaming ASR. We present the dual causal/non-causal self-attention architecture, which in contrast to restricted self-attention prevents the overall context to grow beyond the look-ahead of a single layer.
  arXiv  Detail & Related papers  (2021-07-02T20:56:13Z)
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search [55.164053971213576]
  convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead. Structured (channel) pruning is usually applied to reduce the model redundancy while preserving the network structure. Existing structured pruning methods require hand-crafted rules which may lead to tremendous pruning space.
  arXiv  Detail & Related papers  (2020-11-04T07:43:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.