Deep Sparse Conformer for Speech Recognition

• URL: http://arxiv.org/abs/2209.00260v1
• Date: Thu, 1 Sep 2022 06:56:11 GMT
• Title: Deep Sparse Conformer for Speech Recognition
• Abstract summary: Conformer has achieved impressive results in Automatic Speech Recognition. We adapt a sparse self-attention mechanism with $mathcalO(LtextlogL)$ in time complexity and memory usage. On the Japanese CSJ-500h dataset, this deep sparse Conformer achieves respectively CERs of 5.52%, 4.03% and 4.50%.
• Score: 1.4649095013539173
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Conformer has achieved impressive results in Automatic Speech Recognition (ASR) by leveraging transformer's capturing of content-based global interactions and convolutional neural network's exploiting of local features. In Conformer, two macaron-like feed-forward layers with half-step residual connections sandwich the multi-head self-attention and convolution modules followed by a post layer normalization. We improve Conformer's long-sequence representation ability in two directions, \emph{sparser} and \emph{deeper}. We adapt a sparse self-attention mechanism with $\mathcal{O}(L\text{log}L)$ in time complexity and memory usage. A deep normalization strategy is utilized when performing residual connections to ensure our training of hundred-level Conformer blocks. On the Japanese CSJ-500h dataset, this deep sparse Conformer achieves respectively CERs of 5.52\%, 4.03\% and 4.50\% on the three evaluation sets and 4.16\%, 2.84\% and 3.20\% when ensembling five deep sparse Conformer variants from 12 to 16, 17, 50, and finally 100 encoder layers.

Related papers

• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory [1.081571058570587]
  AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism.<n>We introduce DODOCO to test two assumptions about the workload.<n> scaling EP changes the per-expert max/mean token ratio by at most 5% within every architecture's measurable range.
  arXiv  Detail & Related papers  (2026-05-20T10:14:00Z)
• Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Prediction [9.96771578356719]
  Existing approaches rely on increasingly complex network architectures to regress depth maps.<n>We propose a multilevel conditional random field (CRF) model built upon the Swin Transformer backbone.<n>Our method achieves state-of-the-art performance, reducing Abs Rel to 0.088 ($-$7.4%) and RMSE to 0.316 ($-$5.4%) on NYU Depth v2, while attaining near-perfect threshold accuracy.
  arXiv  Detail & Related papers  (2026-04-03T07:59:26Z)
• MergeDNA: Context-aware Genome Modeling with Dynamic Tokenization through Token Merging [65.07273789940116]
  This paper introduces a hierarchical architecture that jointly optimize a dynamic genomic tokenizer and latent Transformers with context-aware pre-training tasks.<n> MergeDNA achieves superior performance on three popular DNA benchmarks and several multi-omics tasks with fine-tuning or zero-shot evaluation.
  arXiv  Detail & Related papers  (2025-11-17T19:27:41Z)
• Automated Circuit Interpretation via Probe Prompting [0.0]
  We present probe prompting, an automated pipeline that transforms attribution graphs into compact, interpretable subgraphs.<n>Across five prompts, probe-prompted subgraphs preserve high explanatory coverage while compressing complexity.<n>Compared to geometric clustering baselines, concept-aligned groups exhibit higher behavioral coherence.
  arXiv  Detail & Related papers  (2025-11-10T11:53:36Z)
• Arithmetic-Mean $μ$P for Modern Architectures: A Unified Learning-Rate Scale for CNNs and ResNets [9.94514344279733]
  Arithmetic-Mean $mu$P constrains not each individual layer but the network-wide average one-step pre-activation second moment to a constant scale.<n>We prove that, for one- and two-dimensional convolutional networks, the maximal-update learning rate satisfies $etastar(L)propto L-3/2$; with zero padding, boundary effects are constant-level as $Ngg k$.
  arXiv  Detail & Related papers  (2025-10-05T19:22:50Z)
• QwenLong-CPRS: Towards $\infty$-LLMs with Dynamic Context Optimization [70.3105638352827]
  QwenLong-CPRS is a context compression framework designed for explicit long-context optimization.<n>QwenLong-CPRS achieves 21.59$times$ context compression alongside 19.15-point average performance gains.
  arXiv  Detail & Related papers  (2025-05-23T16:47:00Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• Attention Enhanced Citrinet for Speech Recognition [1.4649095013539173]
  We introduce multi-head attentions together with feed-forward networks in the convolution module in Citrinet blocks. For speeding up, we remove 8 convolution layers in each attention-enhanced Citrinet block and reduce 23 blocks to 13. Experiments show that the attention-enhanced Citrinet with less layers and blocks and converges faster with lower character error rates.
  arXiv  Detail & Related papers  (2022-09-01T06:59:50Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)
• CGS-Net: Aggregating Colour, Geometry and Semantic Features for Large-Scale Indoor Place Recognition [6.156387608994791]
  We describe an approach to large-scale indoor place recognition that aggregates low-level colour and geometric features with high-level semantic features. We use a deep learning network that takes in RGB point clouds and extracts local features with five 3-D kernel point convolutional layers. We specifically train the KPConv layers on the semantic segmentation task to ensure that the extracted local features are semantically meaningful.
  arXiv  Detail & Related papers  (2022-02-04T10:51:25Z)
• Augmenting Convolutional networks with attention-based aggregation [55.97184767391253]
  We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth) It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption.
  arXiv  Detail & Related papers  (2021-12-27T14:05:41Z)
• Efficient conformer: Progressive downsampling and grouped attention for automatic speech recognition [2.6346614942667235]
  We study how to reduce the Conformer architecture complexity with a limited computing budget. We introduce progressive downsampling to the Conformer encoder and propose a novel attention mechanism named grouped attention. Within the same computing budget, the proposed architecture achieves better performances with faster training and decoding.
  arXiv  Detail & Related papers  (2021-08-31T07:48:06Z)
• Do End-to-end Stereo Algorithms Under-utilize Information? [7.538482310185133]
  We show how deep adaptive filtering and differentiable semi-global aggregation can be integrated in 2D and 3D convolutional networks for end-to-end stereo matching. The improvements are due to utilizing RGB information from the images as a signal to dynamically guide the matching process.
  arXiv  Detail & Related papers  (2020-10-14T18:32:39Z)
• Regularized Densely-connected Pyramid Network for Salient Instance Segmentation [73.17802158095813]
  We propose a new pipeline for end-to-end salient instance segmentation (SIS) To better use the rich feature hierarchies in deep networks, we propose the regularized dense connections. A novel multi-level RoIAlign based decoder is introduced to adaptively aggregate multi-level features for better mask predictions.
  arXiv  Detail & Related papers  (2020-08-28T00:13:30Z)
• Second-Order Provable Defenses against Adversarial Attacks [63.34032156196848]
  We show that if the eigenvalues of the network are bounded, we can compute a certificate in the $l$ norm efficiently using convex optimization. We achieve certified accuracy of 5.78%, and 44.96%, and 43.19% on 2,59% and 4BP-based methods respectively.
  arXiv  Detail & Related papers  (2020-06-01T05:55:18Z)
• Training Binary Neural Networks with Real-to-Binary Convolutions [52.91164959767517]
  We show how to train binary networks to within a few percent points of the full precision counterpart. We show how to build a strong baseline, which already achieves state-of-the-art accuracy. We show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2020-03-25T17:54:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.