Dynamic Clone Transformer for Efficient Convolutional Neural Netwoks

• URL: http://arxiv.org/abs/2106.06778v1
• Date: Sat, 12 Jun 2021 13:42:28 GMT
• Title: Dynamic Clone Transformer for Efficient Convolutional Neural Netwoks
• Authors: Longqing Ye
• Abstract summary: We introduce a novel concept termed multi-path fully connected pattern (MPFC) to rethink the interdependencies of topology pattern, accuracy and efficiency for ConvNets. Inspired by MPFC, we propose a dual-branch module named dynamic clone transformer (DCT) where one branch generates multiple replicas from inputs and another branch reforms those clones through a series of difference vectors conditional on inputs itself to produce more variants.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Convolutional networks (ConvNets) have shown impressive capability to solve various vision tasks. Nevertheless, the trade-off between performance and efficiency is still a challenge for a feasible model deployment on resource-constrained platforms. In this paper, we introduce a novel concept termed multi-path fully connected pattern (MPFC) to rethink the interdependencies of topology pattern, accuracy and efficiency for ConvNets. Inspired by MPFC, we further propose a dual-branch module named dynamic clone transformer (DCT) where one branch generates multiple replicas from inputs and another branch reforms those clones through a series of difference vectors conditional on inputs itself to produce more variants. This operation allows the self-expansion of channel-wise information in a data-driven way with little computational cost while providing sufficient learning capacity, which is a potential unit to replace computationally expensive pointwise convolution as an expansion layer in the bottleneck structure.

Related papers

• Multiset Transformer: Advancing Representation Learning in Persistence Diagrams [11.512742322405906]
  Multiset Transformer is a neural network that utilizes attention mechanisms specifically designed for multisets as inputs. The architecture integrates multiset-enhanced attentions with a pool-decomposition scheme, allowing multiplicities to be preserved across equivariant layers. Experimental results demonstrate that the Multiset Transformer outperforms existing neural network methods in the realm of persistence diagram representation learning.
  arXiv  Detail & Related papers  (2024-11-22T01:38:47Z)
• Flexible Coded Distributed Convolution Computing for Enhanced Fault Tolerance and Numerical Stability in Distributed CNNs [26.347141131107172]
  This paper introduces the Flexible Coded Distributed Convolution Computing framework. It enhances fault tolerance and numerical stability in distributed CNNs. Empirical results validate the framework's effectiveness in computational efficiency, fault tolerance, and scalability.
  arXiv  Detail & Related papers  (2024-11-03T14:05:29Z)
• Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution [6.857919231112562]
  Window-based transformers have demonstrated outstanding performance in super-resolution tasks. They exhibit higher computational complexity and inference latency than convolutional neural networks. We construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet)
  arXiv  Detail & Related papers  (2024-09-26T07:24:09Z)
• RWKV: Reinventing RNNs for the Transformer Era [54.716108899349614]
  We propose a novel model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers.
  arXiv  Detail & Related papers  (2023-05-22T13:57:41Z)
• Sequence Modeling with Multiresolution Convolutional Memory [27.218134279968062]
  We build a new building block for sequence modeling called a MultiresLayer. The key component of our model is the multiresolution convolution, capturing multiscale trends in the input sequence. Our model yields state-of-the-art performance on a number of sequence classification and autoregressive density estimation tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:50:54Z)
• PECAN: A Product-Quantized Content Addressable Memory Network [6.530758154165138]
  The filtering and linear transform are realized solely with product quantization (PQ) This results in a natural implementation via content addressable memory (CAM) Experiments confirm the feasibility of such Product-Quantized Content Addressable Memory Network (PECAN)
  arXiv  Detail & Related papers  (2022-08-13T08:33:56Z)
• OneDConv: Generalized Convolution For Transform-Invariant Representation [76.15687106423859]
  We propose a novel generalized one dimension convolutional operator (OneDConv) It dynamically transforms the convolution kernels based on the input features in a computationally and parametrically efficient manner. It improves the robustness and generalization of convolution without sacrificing the performance on common images.
  arXiv  Detail & Related papers  (2022-01-15T07:44:44Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• PnP-DETR: Towards Efficient Visual Analysis with Transformers [146.55679348493587]
  Recently, DETR pioneered the solution vision tasks with transformers, it directly translates the image feature map into the object result. Recent transformer-based image recognition model andTT show consistent efficiency gain.
  arXiv  Detail & Related papers  (2021-09-15T01:10:30Z)
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias [76.16156833138038]
  We propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context. In each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network.
  arXiv  Detail & Related papers  (2021-06-07T05:31:06Z)
• Diverse Branch Block: Building a Convolution as an Inception-like Unit [123.59890802196797]
  We propose a universal building block of Convolutional Neural Network (ConvNet) to improve the performance without any inference-time costs. The Diverse Branch Block (DBB) enhances the representational capacity of a single convolution by combining diverse branches of different scales and complexities. After training, a DBB can be equivalently converted into a single conv layer for deployment.
  arXiv  Detail & Related papers  (2021-03-24T18:12:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.