DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR

• URL: http://arxiv.org/abs/2306.08175v2
• Date: Fri, 1 Mar 2024 21:25:16 GMT
• Title: DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR
• Abstract summary: We propose the integration of a novel dynamic contextual carry-over mechanism in a state-of-the-art unified ASR system. Our proposed dynamic context Conformer (DCTX-Conformer) utilizes a non-overlapping contextual carry-over mechanism. We outperform the SOTA by a relative 25.0% word error rate, with a negligible latency impact due to the additional context embeddings.
• Score: 20.42366884075422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformer-based end-to-end models have become ubiquitous these days and are commonly used in both streaming and non-streaming automatic speech recognition (ASR). Techniques like dual-mode and dynamic chunk training helped unify streaming and non-streaming systems. However, there remains a performance gap between streaming with a full and limited past context. To address this issue, we propose the integration of a novel dynamic contextual carry-over mechanism in a state-of-the-art (SOTA) unified ASR system. Our proposed dynamic context Conformer (DCTX-Conformer) utilizes a non-overlapping contextual carry-over mechanism that takes into account both the left context of a chunk and one or more preceding context embeddings. We outperform the SOTA by a relative 25.0% word error rate, with a negligible latency impact due to the additional context embeddings.

Related papers

• Enabling Streaming User Transcription in Full-Duplex Speech-to-Speech Models [54.83270723528963]
  We propose a method to add streaming capabilities to an existing S2S model.<n>Our approach requires minimal additional parameters and no significant changes to the base S2S model.
  arXiv  Detail & Related papers  (2026-09-14T15:44:18Z)
• TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution [52.96050253456634]
  TRaM-VSR is a framework for adaptive token allocation using context-aware video priors and network-levels.<n>It is shown that TRaM-VSR accelerates inference significantly while preserving state-of-the-art reconstruction quality and robust temporal consistency.
  arXiv  Detail & Related papers  (2026-07-24T11:57:26Z)
• Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization [35.546900956309294]
  We present a Unified ASR framework for Transducer (RNNT) training that supports both offline and streaming decoding within a single model.<n>Experiments show that the proposed approach improves streaming accuracy at low latency while preserving offline performance and scaling to larger model sizes and training datasets.
  arXiv  Detail & Related papers  (2026-04-21T04:43:03Z)
• SENS-ASR: Semantic Embedding injection in Neural-transducer for Streaming Automatic Speech Recognition [3.0406449751520754]
  We present SENS-ASR, an approach to enhance the transcription quality of Streaming-ASR by reinforcing the acoustic information with semantic information.<n>Experiments on standard datasets show that SENS-ASR significantly improves the Word Error Rate on small-chunk streaming scenarios.
  arXiv  Detail & Related papers  (2026-02-17T07:18:07Z)
• OneLive: Dynamically Unified Generative Framework for Live-Streaming Recommendation [49.95897358060393]
  We propose OneLive, a dynamically unified generative recommendation framework tailored for live-streaming scenario.<n>OneLive integrates four key components: (i) A Dynamic Tokenizer that continuously encodes evolving real-time live content fused with behavior signal through residual quantization; (ii) A Time-Aware Gated Attention mechanism that explicitly models temporal dynamics for timely decision making; (iii) An efficient decoder-only generative architecture enhanced with Sequential MTP and QK Norm for stable training and accelerated inference.
  arXiv  Detail & Related papers  (2026-02-09T12:56:39Z)
• Real-Time Inference for Distributed Multimodal Systems under Communication Delay Uncertainty [37.15356899831919]
  Connected cyber-physical systems perform inference based on real-time inputs from multiple data streams.<n>We propose a novel neuro-inspired non-blocking inference paradigm that employs adaptive temporal windows of integration.<n>Our framework achieves robust real-time inference with finer-grained control over the accuracy-latency tradeoff.
  arXiv  Detail & Related papers  (2025-11-20T10:48:54Z)
• Unifying Streaming and Non-streaming Zipformer-based ASR [14.226219579716629]
  We present a unified framework that trains a single end-to-end ASR model for both streaming and non-streaming applications.<n>We propose to use dynamic right-context through the chunked attention masking in the training of zipformer-based ASR models.<n>We analyze the effect of varying the number of right-context frames on accuracy and latency of the streaming ASR models.
  arXiv  Detail & Related papers  (2025-06-17T11:52:41Z)
• FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers [63.788600404496115]
  FullDiT2 is an efficient in-context conditioning framework for general controllability in both video generation and editing tasks.<n>FullDiT2 achieves significant computation reduction and 2-3 times speedup in averaged time cost per diffusion step.
  arXiv  Detail & Related papers  (2025-06-04T17:57:09Z)
• Dynamic Context-Aware Streaming Pretrained Language Model For Inverse Text Normalization [0.19791587637442667]
  Inverse Text Normalization (ITN) is crucial for converting spoken Automatic Speech Recognition (ASR) outputs into well-formatted written text.<n>We introduce a streaming pretrained language model for ITN, leveraging pretrained linguistic representations for improved robustness.<n>Our method achieves accuracy comparable to non-streaming ITN and surpasses existing streaming ITN models on a Vietnamese dataset.
  arXiv  Detail & Related papers  (2025-05-30T05:41:03Z)
• Zero-Shot Streaming Text to Speech Synthesis with Transducer and Auto-Regressive Modeling [76.23539797803681]
  Existing methods primarily use a look mechanism, relying on future text to achieve natural streaming speech synthesis.<n>We propose LE, a streaming framework for generating high-quality speech frame-by-frame.<n> Experimental results suggest that the LE outperforms current streaming TTS methods and achieves comparable performance over sentence-level TTS systems.
  arXiv  Detail & Related papers  (2025-05-26T08:25:01Z)
• Neural Video Compression with Context Modulation [9.875413481663742]
  In this paper, we address the limitation by modulating the temporal context with the reference frame in two steps.<n>We achieve on average 22.7% reduction over the advanced traditional video H.266/VVC, and offer an average 10.1% saving over the previous state-of-the-art NVC DCVC-FM.
  arXiv  Detail & Related papers  (2025-05-20T15:57:09Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech. The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture. To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• Streaming Sequence Transduction through Dynamic Compression [55.0083843520833]
  We introduce STAR (Stream Transduction with Anchor Representations), a novel Transformer-based model designed for efficient sequence-to-sequence transduction over streams. STAR dynamically segments input streams to create compressed anchor representations, achieving nearly lossless compression (12x) in Automatic Speech Recognition (ASR) STAR demonstrates superior segmentation and latency-quality trade-offs in simultaneous speech-to-text tasks, optimizing latency, memory footprint, and quality.
  arXiv  Detail & Related papers  (2024-02-02T06:31:50Z)
• CUSIDE: Chunking, Simulating Future Context and Decoding for Streaming ASR [17.999404155015647]
  We propose a new framework - Chunking, Simulating Future Context and Decoding (CUSIDE) for streaming speech recognition. A new simulation module is introduced to simulate the future contextual frames, without waiting for future context. Experiments show that, compared to using real future frames as right context, using simulated future context can drastically reduce latency while maintaining recognition accuracy.
  arXiv  Detail & Related papers  (2022-03-31T02:28:48Z)
• End-to-End Active Speaker Detection [58.7097258722291]
  We propose an end-to-end training network where feature learning and contextual predictions are jointly learned. We also introduce intertemporal graph neural network (iGNN) blocks, which split the message passing according to the main sources of context in the ASD problem. Experiments show that the aggregated features from the iGNN blocks are more suitable for ASD, resulting in state-of-the art performance.
  arXiv  Detail & Related papers  (2022-03-27T08:55:28Z)
• Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models [57.20432226304683]
  Non-autoregressive (NAR) modeling has gained more and more attention in speech processing. We propose a novel end-to-end streaming NAR speech recognition system. We show that the proposed method improves online ASR recognition in low latency conditions.
  arXiv  Detail & Related papers  (2021-07-20T11:42:26Z)
• Dual Causal/Non-Causal Self-Attention for Streaming End-to-End Speech Recognition [58.69803243323346]
  Attention-based end-to-end automatic speech recognition (ASR) systems have recently demonstrated state-of-the-art results for numerous tasks. However, the application of self-attention and attention-based encoder-decoder models remains challenging for streaming ASR. We present the dual causal/non-causal self-attention architecture, which in contrast to restricted self-attention prevents the overall context to grow beyond the look-ahead of a single layer.
  arXiv  Detail & Related papers  (2021-07-02T20:56:13Z)
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling [76.43479696760996]
  We propose a unified framework, Dual-mode ASR, to train a single end-to-end ASR model with shared weights for both streaming and full-context speech recognition. We show that the latency and accuracy of streaming ASR significantly benefit from weight sharing and joint training of full-context ASR.
  arXiv  Detail & Related papers  (2020-10-12T21:12:56Z)
• Conv-Transformer Transducer: Low Latency, Low Frame Rate, Streamable End-to-End Speech Recognition [8.046120977786702]
  Transformer has achieved competitive performance against state-of-the-art end-to-end models in automatic speech recognition (ASR) The original Transformer, with encoder-decoder architecture, is only suitable for offline ASR. We show that this architecture, named Conv-Transformer Transducer, achieves competitive performance on LibriSpeech dataset (3.6% WER on test-clean) without external language models.
  arXiv  Detail & Related papers  (2020-08-13T08:20:02Z)
• Streaming automatic speech recognition with the transformer model [59.58318952000571]
  We propose a transformer based end-to-end ASR system for streaming ASR. We apply time-restricted self-attention for the encoder and triggered attention for the encoder-decoder attention mechanism. Our proposed streaming transformer architecture achieves 2.8% and 7.2% WER for the "clean" and "other" test data of LibriSpeech.
  arXiv  Detail & Related papers  (2020-01-08T18:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.