DCT Perceptron Layer: A Transform Domain Approach for Convolution Layer

• URL: http://arxiv.org/abs/2211.08577v1
• Date: Tue, 15 Nov 2022 23:44:56 GMT
• Title: DCT Perceptron Layer: A Transform Domain Approach for Convolution Layer
• Authors: Hongyi Pan, Xin Zhu, Salih Atici, Ahmet Enis Cetin
• Abstract summary: We propose a novel Discrete Cosine Transform (DCT)-based neural network layer which we call DCT-perceptron. Convolutional filtering operations are performed in the DCT domain using element-wise multiplications. The DCT-perceptron layer reduces the number of parameters and multiplications significantly.
• Score: 3.506018346865459
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we propose a novel Discrete Cosine Transform (DCT)-based neural network layer which we call DCT-perceptron to replace the $3\times3$ Conv2D layers in the Residual neural Network (ResNet). Convolutional filtering operations are performed in the DCT domain using element-wise multiplications by taking advantage of the Fourier and DCT Convolution theorems. A trainable soft-thresholding layer is used as the nonlinearity in the DCT perceptron. Compared to ResNet's Conv2D layer which is spatial-agnostic and channel-specific, the proposed layer is location-specific and channel-specific. The DCT-perceptron layer reduces the number of parameters and multiplications significantly while maintaining comparable accuracy results of regular ResNets in CIFAR-10 and ImageNet-1K. Moreover, the DCT-perceptron layer can be inserted with a batch normalization layer before the global average pooling layer in the conventional ResNets as an additional layer to improve classification accuracy.

Related papers

• TCCT-Net: Two-Stream Network Architecture for Fast and Efficient Engagement Estimation via Behavioral Feature Signals [58.865901821451295]
  We present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture. To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer. In parallel, to efficiently extract rich patterns from the temporal-frequency domain, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form.
  arXiv  Detail & Related papers  (2024-04-15T06:01:48Z)
• Image Reconstruction for Accelerated MR Scan with Faster Fourier Convolutional Neural Networks [87.87578529398019]
  Partial scan is a common approach to accelerate Magnetic Resonance Imaging (MRI) data acquisition in both 2D and 3D settings. We propose a novel convolutional operator called Faster Fourier Convolution (FasterFC) to replace the two consecutive convolution operations. A 2D accelerated MRI method, FasterFC-End-to-End-VarNet, which uses FasterFC to improve the sensitivity maps and reconstruction quality. A 3D accelerated MRI method called FasterFC-based Single-to-group Network (FAS-Net) that utilizes a single-to-group algorithm to guide k-space domain reconstruction
  arXiv  Detail & Related papers  (2023-06-05T13:53:57Z)
• Deep Convolutional Tables: Deep Learning without Convolutions [12.069186324544347]
  We propose a novel formulation of deep networks that do not use dot-product neurons and rely on a hierarchy of voting tables instead. Deep CT networks have been experimentally shown to have accuracy comparable to that of CNNs of similar architectures.
  arXiv  Detail & Related papers  (2023-04-23T17:49:21Z)
• Multichannel Orthogonal Transform-Based Perceptron Layers for Efficient ResNets [2.829818195105779]
  We propose a set of transform-based neural network layers as an alternative to the $3times3$ Conv2D layers in CNNs. The proposed layers can be implemented based on transforms such as the Discrete Cosine Transform (DCT), Hadamard transform (HT), and biorthogonal Block Wavelet Transform (BWT)
  arXiv  Detail & Related papers  (2023-03-13T01:07:32Z)
• WLD-Reg: A Data-dependent Within-layer Diversity Regularizer [98.78384185493624]
  Neural networks are composed of multiple layers arranged in a hierarchical structure jointly trained with a gradient-based optimization. We propose to complement this traditional 'between-layer' feedback with additional 'within-layer' feedback to encourage the diversity of the activations within the same layer. We present an extensive empirical study confirming that the proposed approach enhances the performance of several state-of-the-art neural network models in multiple tasks.
  arXiv  Detail & Related papers  (2023-01-03T20:57:22Z)
• CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution [55.50793823060282]
  We propose a novel Content-Aware Dynamic Quantization (CADyQ) method for image super-resolution (SR) networks. CADyQ allocates optimal bits to local regions and layers adaptively based on the local contents of an input image. The pipeline has been tested on various SR networks and evaluated on several standard benchmarks.
  arXiv  Detail & Related papers  (2022-07-21T07:50:50Z)
• Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks [7.906608953906891]
  Convolution has been the core operation of modern deep neural networks. We propose to use binary block Walsh-Hadamard transform (WHT) instead of the Fourier transform. We use WHT-based binary layers to replace some of the regular convolution layers in deep neural networks.
  arXiv  Detail & Related papers  (2022-01-07T23:52:41Z)
• Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution [80.93870349019332]
  We propose a novel HSISR method that uses Transformer instead of CNN to learn the prior of HSIs. Specifically, we first use the gradient algorithm to solve the HSISR model, and then use an unfolding network to simulate the iterative solution processes.
  arXiv  Detail & Related papers  (2021-11-27T15:38:57Z)
• Convolutional Neural Network Compression through Generalized Kronecker Product Decomposition [2.4240083226965115]
  We compress layers by generalizing the Kronecker Product Decomposition to apply to multidimensionals, leading to the Generalized Kronecker Product Decomposition(GKPD) Our approach yields a plug-and-play module that can be used as a drop-in replacement for any convolutional layer.
  arXiv  Detail & Related papers  (2021-09-29T20:45:08Z)
• DO-Conv: Depthwise Over-parameterized Convolutional Layer [66.46704754669169]
  We propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs.
  arXiv  Detail & Related papers  (2020-06-22T06:57:10Z)
• Res-CR-Net, a residual network with a novel architecture optimized for the semantic segmentation of microscopy images [0.5363346028859919]
  Res-CR-Net is a type of Deep Neural Network (DNN) that features residual blocks with either a bundle of separable atrous convolutions with different dilation rates or a convolutional LSTM. The number of filters used in each residual block and the number of blocks are the only hyper parameters that need to be modified in order to optimize the network training for a variety of different microscopy images.
  arXiv  Detail & Related papers  (2020-04-14T21:21:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.