SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution

• URL: http://arxiv.org/abs/2211.11419v4
• Date: Sun, 4 Feb 2024 08:03:23 GMT
• Title: SSCFormer: Push the Limit of Chunk-wise Conformer for Streaming ASR Using Sequentially Sampled Chunks and Chunked Causal Convolution
• Abstract summary: We propose SSCFormer to push the limit of chunk-wise Conformer for streaming ASR. We show that an End-to-End (E2E) CER 5.33% can achieve, which even outperforms a strong time-restricted baseline U2.
• Score: 10.896823447820232
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Currently, the chunk-wise schemes are often used to make Automatic Speech Recognition (ASR) models to support streaming deployment. However, existing approaches are unable to capture the global context, lack support for parallel training, or exhibit quadratic complexity for the computation of multi-head self-attention (MHSA). On the other side, the causal convolution, no future context used, has become the de facto module in streaming Conformer. In this paper, we propose SSCFormer to push the limit of chunk-wise Conformer for streaming ASR using the following two techniques: 1) A novel cross-chunks context generation method, named Sequential Sampling Chunk (SSC) scheme, to re-partition chunks from regular partitioned chunks to facilitate efficient long-term contextual interaction within local chunks. 2)The Chunked Causal Convolution (C2Conv) is designed to concurrently capture the left context and chunk-wise future context. Evaluations on AISHELL-1 show that an End-to-End (E2E) CER 5.33% can achieve, which even outperforms a strong time-restricted baseline U2. Moreover, the chunk-wise MHSA computation in our model enables it to train with a large batch size and perform inference with linear complexity.

Related papers

• ConvMem: Convolutional Memory for Long-Context Reasoning [18.597196274231553]
  ConvMem is a training-free, highly parallelizable framework that reformulates long-context reasoning as a hierarchical convolution.<n>We show that ConvMem outperforms training-free baselines and avoids the risk of overfitting to parametric priors on out-of-distribution tasks.
  arXiv  Detail & Related papers  (2026-09-09T16:53:02Z)
• An Efficient vLLM-Based Inference Pipeline for Unified Audio Understanding and Generation [63.50393485685279]
  We present a vLLM-based inference pipeline for unified speech understanding and generation.<n>We overcome the shared intuition-Free Guidance (CFG) halves throughput.<n>By co-scheduling paired conditional and unconditional requests within a continuous batch, our CFG implementation sustains 80% of non-CFG throughput.
  arXiv  Detail & Related papers  (2026-07-02T12:55:11Z)
• MURMUR: An Efficient Inference System for Long-Form ASR [5.721402685889177]
  Long-form automatic speech recognition requires both high accuracy and low latency.<n>Chunk-based pipelines process audio in parallel windows for low latency, but lose cross-context context.<n>Long-context models resolve everything in a single pass for better accuracy, but are an order of magnitude slower.<n>We propose Murmur, an inference system that overcomes this trade-off by operating at two levels.
  arXiv  Detail & Related papers  (2026-05-31T22:54:57Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• Efficient Learned Data Compression via Dual-Stream Feature Decoupling [11.752785837648199]
  We propose a Dual-Stream Multi-Scale Decoupler that disentangles local and global contexts to replace deep serial processing with shallow parallel streams.<n>Our method achieves state-of-the-art performance in both compression ratio and throughput, while maintaining the lowest latency and memory usage.
  arXiv  Detail & Related papers  (2026-04-08T16:05:24Z)
• Stream2LLM: Overlap Context Streaming and Prefill for Reduced Time-to-First-Token (TTFT) [2.8350851063316873]
  Streaming context incrementally--overlapping retrieval with inference--can mitigate this latency.<n>We present Stream2LLM, a streaming-aware LLM serving system for concurrent prefill-decode disaggregated deployments.
  arXiv  Detail & Related papers  (2026-03-29T06:49:12Z)
• Spectral-Window Hybrid (SWH) [0.0]
  Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity.<n>We propose the textbfSpectral-Window Hybrid (SWH), an architecture that decouples sequence modeling into two textitparallel streams.<n>We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences.
  arXiv  Detail & Related papers  (2026-01-04T00:31:36Z)
• TNT: Improving Chunkwise Training for Test-Time Memorization [62.78875147721906]
  Recurrent neural networks (RNNs) with deep test-time memorization modules, such as Titans and TTT, represent a promising, linearly-scaling paradigm distinct from Transformers.<n>We introduce TNT, a novel training paradigm that decouples training efficiency from inference performance through a two-stage process.<n>TNT achieves a substantial acceleration in training speed-up to 17 times faster than the most accurate baseline configuration.
  arXiv  Detail & Related papers  (2025-11-10T17:45:09Z)
• AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding [35.10915929939651]
  Test-time scaling (TTS) boosts LLM reasoning via long chain-of-thought (CoT)<n> KV-cache growth amplifies the memory-bound bottleneck of LLM decoding.<n>We propose AsyncSpade, an asynchronous framework for efficient TTS built on two core components.
  arXiv  Detail & Related papers  (2025-10-08T19:36:11Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Trajectory-aware Shifted State Space Models for Online Video Super-Resolution [57.87099307245989]
  This paper presents a novel online VSR method based on Trajectory-aware Shifted SSMs (TS-Mamba)<n>TS-Mamba first constructs the trajectories within a video to select the most similar tokens from the previous frames.<n>Our TS-Mamba achieves state-of-the-art performance in most cases and over 22.7% reduction complexity (in MACs)
  arXiv  Detail & Related papers  (2025-08-14T08:42:15Z)
• Next Tokens Denoising for Speech Synthesis [51.320443764269726]
  Dragon-FM is a novel text-to-speech (TTS) design that unifies AR and flow-matching.<n>It processes 48 kHz audio tokens in chunks at a compact rate of 12.5 tokens per second.<n>Experiments on podcast datasets demonstrate its capability to efficiently generate high-quality zero-shot podcasts.
  arXiv  Detail & Related papers  (2025-07-30T15:03:36Z)
• CCLSTM: Coupled Convolutional Long-Short Term Memory Network for Occupancy Flow Forecasting [0.0]
  We propose textbfCoupled Convolutional LSTM (CTM), a lightweight, end-to-end trainable architecture based solely on convolutional operations.<n>CTM achieves state-of-the-art performance on occupancy flow metrics and, as of this submission, ranks (textst) in all metrics on the 2024 Occupancy and Flow Prediction Challenge leaderboard.
  arXiv  Detail & Related papers  (2025-06-06T14:38:55Z)
• Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers [58.98923344096319]
  REFORM is a novel inference framework that efficiently handles long contexts through a two-phase approach.<n>It achieves over 50% and 27% performance gains on RULER and BABILong respectively at 1M context length.<n>It also outperforms baselines on Infinite-Bench and MM-NIAH, demonstrating flexibility across diverse tasks and domains.
  arXiv  Detail & Related papers  (2025-06-01T23:49:14Z)
• Training Long-Context LLMs Efficiently via Chunk-wise Optimization [60.05884946552877]
  We present textitSequential Chunk-wise Optimization (SeCO), a memory-efficient training paradigm that partitions lengthy inputs into manageable chunks.<n>We also introduce textitSparse Chunk-wise Optimization (SpaCO), which reduces computational overhead by selectively propagating gradients to specific chunks.<n>SpaCO decouples the computational cost of backpropagation from the context length, enabling training time to gradually converge to inference time as sequences become longer.
  arXiv  Detail & Related papers  (2025-05-22T14:11:34Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs) StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks. Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)
• Tractable Offline Learning of Regular Decision Processes [50.11277112628193]
  This work studies offline Reinforcement Learning (RL) in a class of non-Markovian environments called Regular Decision Processes (RDPs) Ins, the unknown dependency of future observations and rewards from the past interactions can be captured experimentally. Many algorithms first reconstruct this unknown dependency using automata learning techniques.
  arXiv  Detail & Related papers  (2024-09-04T14:26:58Z)
• KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches [52.02764371205856]
  Long context capability is a crucial competency for large language models (LLMs) This work provides a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:59:47Z)
• Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling [49.215957313126324]
  Recently developed generative methods, including invertible rescaling network (IRN) based and generative adversarial network (GAN) based methods, have demonstrated exceptional performance in image rescaling. However, IRN-based methods tend to produce over-smoothed results, while GAN-based methods easily generate fake details. We propose Boundary-aware Decoupled Flow Networks (BDFlow) to generate realistic and visually pleasing results.
  arXiv  Detail & Related papers  (2024-05-05T14:05:33Z)
• CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning [59.88924847995279]
  We propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF. To reduce the distribution discrepancy, we develop the cross-modal match module. CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks.
  arXiv  Detail & Related papers  (2024-03-12T04:04:38Z)
• C2FAR: Coarse-to-Fine Autoregressive Networks for Precise Probabilistic Forecasting [6.046449209642488]
  coarse-to-fine autoregressive networks (C2FAR) are presented. C2FAR represents values with exponentially higher precision, for only a linear increase in complexity. We use C2FAR for probabilistic forecasting via a recurrent neural network, thus modeling time series autoregressively in both space and time.
  arXiv  Detail & Related papers  (2023-12-22T16:37:52Z)
• StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences [31.210626775505407]
  Occlusions between consecutive frames have long posed a significant challenge in optical flow estimation. We present a Streamlined In-batch Multi-frame (SIM) pipeline tailored to video input, attaining a similar level of time efficiency to two-frame networks. StreamFlow not only excels in terms of performance on challenging KITTI and Sintel datasets, with particular improvement in occluded areas.
  arXiv  Detail & Related papers  (2023-11-28T07:53:51Z)
• DCTX-Conformer: Dynamic context carry-over for low latency unified streaming and non-streaming Conformer ASR [20.42366884075422]
  We propose the integration of a novel dynamic contextual carry-over mechanism in a state-of-the-art unified ASR system. Our proposed dynamic context Conformer (DCTX-Conformer) utilizes a non-overlapping contextual carry-over mechanism. We outperform the SOTA by a relative 25.0% word error rate, with a negligible latency impact due to the additional context embeddings.
  arXiv  Detail & Related papers  (2023-06-13T23:42:53Z)
• Recurrent Attention Networks for Long-text Modeling [14.710722261441822]
  This paper proposes a novel long-document encoding model, Recurrent Attention Network (RAN), to enable the recurrent operation of self-attention. RAN is capable of extracting global semantics in both token-level and document-level representations, making it inherently compatible with both sequential and classification tasks.
  arXiv  Detail & Related papers  (2023-06-12T03:28:33Z)
• Shifted Chunk Encoder for Transformer Based Streaming End-to-End ASR [7.828989755009444]
  We build a single model to achieve the benefits of all the three aspects for streaming E2E ASR. We name the Shifted Chunk Transformer and Conformer as SChunk-Transofromer and SChunk-Conformer, respectively. Experiments show that the SChunk-Transformer and SChunk-Conformer achieve CER 6.43% and 5.77%, respectively.
  arXiv  Detail & Related papers  (2022-03-29T03:02:35Z)
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding [90.77031668988661]
  Cluster-Former is a novel clustering-based sparse Transformer to perform attention across chunked sequences. The proposed framework is pivoted on two unique types of Transformer layer: Sliding-Window Layer and Cluster-Former Layer. Experiments show that Cluster-Former achieves state-of-the-art performance on several major QA benchmarks.
  arXiv  Detail & Related papers  (2020-09-13T22:09:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.