Sequential Routing Framework: Fully Capsule Network-based Speech Recognition

• URL: http://arxiv.org/abs/2007.11747v3
• Date: Thu, 1 Apr 2021 09:09:29 GMT
• Title: Sequential Routing Framework: Fully Capsule Network-based Speech Recognition
• Authors: Kyungmin Lee, Hyunwhan Joe, Hyeontaek Lim, Kwangyoun Kim, Sungsoo Kim, Chang Woo Han, Hong-Gee Kim
• Abstract summary: This paper presents a sequential routing framework to adapt a CapsNet-only structure to sequence-to-sequence recognition. It achieves a 1.1% lower word error rate at 16.9% on the Wall Street Journal corpus. It attains a 0.7% lower phone error rate at 17.5% compared to convolutional neural network-based CTC networks.
• Score: 5.730259752695884
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Capsule networks (CapsNets) have recently gotten attention as a novel neural architecture. This paper presents the sequential routing framework which we believe is the first method to adapt a CapsNet-only structure to sequence-to-sequence recognition. Input sequences are capsulized then sliced by a window size. Each slice is classified to a label at the corresponding time through iterative routing mechanisms. Afterwards, losses are computed by connectionist temporal classification (CTC). During routing, the required number of parameters can be controlled by the window size regardless of the length of sequences by sharing learnable weights across the slices. We additionally propose a sequential dynamic routing algorithm to replace traditional dynamic routing. The proposed technique can minimize decoding speed degradation caused by the routing iterations since it can operate in a non-iterative manner without dropping accuracy. The method achieves a 1.1% lower word error rate at 16.9% on the Wall Street Journal corpus compared to bidirectional long short-term memory-based CTC networks. On the TIMIT corpus, it attains a 0.7% lower phone error rate at 17.5% compared to convolutional neural network-based CTC networks (Zhang et al., 2016).

Related papers

• OrthCaps: An Orthogonal CapsNet with Sparse Attention Routing and Pruning [21.5857226735951]
  Redundancy is a persistent challenge in Capsule Networks (CapsNet) We propose an Orthogonal Capsule Network (OrthCaps) to reduce redundancy, improve routing performance and decrease parameter counts.
  arXiv  Detail & Related papers  (2024-03-20T07:25:24Z)
• LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch [14.911305800463285]
  We propose a novel framework named Layer Adaptive Progressive Pruning (LAPP) LAPP designs an effective and efficient pruning strategy that introduces a learnable threshold for each layer and FLOPs constraints for network. Our method demonstrates superior performance gains over previous compression methods on various datasets and backbone architectures.
  arXiv  Detail & Related papers  (2023-09-25T14:08:45Z)
• Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
  We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter. We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures'' Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
  arXiv  Detail & Related papers  (2022-10-23T18:37:22Z)
• Routing Towards Discriminative Power of Class Capsules [7.347145775695176]
  We propose a routing algorithm that incorporates a regularized quadratic programming problem which can be solved efficiently. We conduct experiments on MNIST, MNIST-Fashion, and CIFAR-10 and show competitive classification results compared to existing capsule networks.
  arXiv  Detail & Related papers  (2021-03-07T05:49:38Z)
• ADOM: Accelerated Decentralized Optimization Method for Time-Varying Networks [124.33353902111939]
  We propose ADOM - an accelerated method for smooth and strongly convex decentralized optimization over time-varying networks. Up to a constant factor, which depends on the network structure only, its communication is the same as that of accelerated Nesterov method.
  arXiv  Detail & Related papers  (2021-02-18T09:37:20Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)
• DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
  This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
  arXiv  Detail & Related papers  (2020-03-30T17:59:18Z)
• Learning Dynamic Routing for Semantic Segmentation [86.56049245100084]
  This paper studies a conceptually new method to alleviate the scale variance in semantic representation, named dynamic routing. The proposed framework generates data-dependent routes, adapting to the scale distribution of each image. To this end, a differentiable gating function, called soft conditional gate, is proposed to select scale transform paths on the fly.
  arXiv  Detail & Related papers  (2020-03-23T17:22:14Z)
• Capsules with Inverted Dot-Product Attention Routing [84.89818784286953]
  We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. Our method improves performance on benchmark datasets such as CIFAR-10 and CIFAR-100. We believe that our work raises the possibility of applying capsule networks to complex real-world tasks.
  arXiv  Detail & Related papers  (2020-02-12T02:09:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.