Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets

• URL: http://arxiv.org/abs/2303.06797v3
• Date: Mon, 22 Apr 2024 22:39:12 GMT
• Title: Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets
• Authors: Hongyi Pan, Emadeldeen Hamdan, Xin Zhu, Salih Atici, Ahmet Enis Cetin,
• Abstract summary: We propose a set of transform-based neural network layers as an alternative to the $3times3$ Conv2D layers in CNNs. The proposed layers can be implemented based on transforms such as the Discrete Cosine Transform (DCT), Hadamard transform (HT), and biorthogonal Block Wavelet Transform (BWT)
• Score: 2.829818195105779
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we propose a set of transform-based neural network layers as an alternative to the $3\times3$ Conv2D layers in Convolutional Neural Networks (CNNs). The proposed layers can be implemented based on orthogonal transforms such as the Discrete Cosine Transform (DCT), Hadamard transform (HT), and biorthogonal Block Wavelet Transform (BWT). Furthermore, by taking advantage of the convolution theorems, convolutional filtering operations are performed in the transform domain using element-wise multiplications. Trainable soft-thresholding layers, that remove noise in the transform domain, bring nonlinearity to the transform domain layers. Compared to the Conv2D layer, which is spatial-agnostic and channel-specific, the proposed layers are location-specific and channel-specific. Moreover, these proposed layers reduce the number of parameters and multiplications significantly while improving the accuracy results of regular ResNets on the ImageNet-1K classification task. Furthermore, they can be inserted with a batch normalization layer before the global average pooling layer in the conventional ResNets as an additional layer to improve classification accuracy.

Related papers

• HoughToRadon Transform: New Neural Network Layer for Features Improvement in Projection Space [83.88591755871734]
  HoughToRadon Transform layer is a novel layer designed to improve the speed of neural networks incorporated with Hough Transform. Our experiments on the open MIDV-500 dataset show that this new approach leads to time savings and achieves state-of-the-art 97.7% accuracy.
  arXiv  Detail & Related papers  (2024-02-05T12:19:16Z)
• WLD-Reg: A Data-dependent Within-layer Diversity Regularizer [98.78384185493624]
  Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. We propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks.
  arXiv  Detail & Related papers  (2023-01-03T20:57:22Z)
• DCT Perceptron Layer: A Transform Domain Approach for Convolution Layer [3.506018346865459]
  We propose a novel Discrete Cosine Transform (DCT)-based neural network layer which we call DCT-perceptron. Convolutional filtering operations are performed in the DCT domain using element-wise multiplications. The DCT-perceptron layer reduces the number of parameters and multiplications significantly.
  arXiv  Detail & Related papers  (2022-11-15T23:44:56Z)
• Learning Invariant Representations for Equivariant Neural Networks Using Orthogonal Moments [9.680414207552722]
  The convolutional layers of standard convolutional neural networks (CNNs) are equivariant to translation. Recently, a new class of CNNs is proposed in which the conventional layers of CNNs are replaced with equivariant convolution, pooling, and batch-normalization layers.
  arXiv  Detail & Related papers  (2022-09-22T11:48:39Z)
• Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks [7.906608953906891]
  Convolution has been the core operation of modern deep neural networks. We propose to use binary block Walsh-Hadamard transform (WHT) instead of the Fourier transform. We use WHT-based binary layers to replace some of the regular convolution layers in deep neural networks.
  arXiv  Detail & Related papers  (2022-01-07T23:52:41Z)
• Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework [16.577482515547793]
  We propose a theoretical framework for orthogonal convolutional layers. Our framework endows high expressive power to various convolutional layers while maintaining their exactity. Our layers are memory and computationally efficient for deep networks compared to previous designs.
  arXiv  Detail & Related papers  (2021-06-16T20:50:59Z)
• Rethinking Skip Connection with Layer Normalization in Transformers and ResNets [49.87919454950763]
  Skip connection is a widely-used technique to improve the performance of deep neural networks. In this work, we investigate how the scale factors in the effectiveness of the skip connection.
  arXiv  Detail & Related papers  (2021-05-15T11:44:49Z)
• Adaptive Convolution Kernel for Artificial Neural Networks [0.0]
  This paper describes a method for training the size of convolutional kernels to provide varying size kernels in a single layer. Experiments compared the proposed adaptive layers to ordinary convolution layers in a simple two-layer network. A segmentation experiment in the Oxford-Pets dataset demonstrated that replacing a single ordinary convolution layer in a U-shaped network with a single 7$times$7 adaptive layer can improve its learning performance and ability to generalize.
  arXiv  Detail & Related papers  (2020-09-14T12:36:50Z)
• Rewiring the Transformer with Depth-Wise LSTMs [55.50278212605607]
  We present a Transformer with depth-wise LSTMs connecting cascading Transformer layers and sub-layers. Experiments with the 6-layer Transformer show significant BLEU improvements in both WMT 14 English-German / French tasks and the OPUS-100 many-to-many multilingual NMT task.
  arXiv  Detail & Related papers  (2020-07-13T09:19:34Z)
• DO-Conv: Depthwise Over-parameterized Convolutional Layer [66.46704754669169]
  We propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs.
  arXiv  Detail & Related papers  (2020-06-22T06:57:10Z)
• Region adaptive graph fourier transform for 3d point clouds [51.193111325231165]
  We introduce the Region Adaptive Graph Fourier Transform (RA-GFT) for compression of 3D point cloud attributes. The RA-GFT achieves better complexity-performance trade-offs than previous approaches.
  arXiv  Detail & Related papers  (2020-03-04T02:47:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.