OrthoNets: Orthogonal Channel Attention Networks

• URL: http://arxiv.org/abs/2311.03071v2
• Date: Tue, 7 Nov 2023 02:23:30 GMT
• Title: OrthoNets: Orthogonal Channel Attention Networks
• Abstract summary: We develop a new type of channel attention mechanism called OrthoNet. Our results imply that an optimal choice of filter is elusive and generalization can be achieved. Our code is publicly available at https://github.com/hady1011/OrthoNets/.
• Score: 9.842943431749191
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Designing an effective channel attention mechanism implores one to find a lossy-compression method allowing for optimal feature representation. Despite recent progress in the area, it remains an open problem. FcaNet, the current state-of-the-art channel attention mechanism, attempted to find such an information-rich compression using Discrete Cosine Transforms (DCTs). One drawback of FcaNet is that there is no natural choice of the DCT frequencies. To circumvent this issue, FcaNet experimented on ImageNet to find optimal frequencies. We hypothesize that the choice of frequency plays only a supporting role and the primary driving force for the effectiveness of their attention filters is the orthogonality of the DCT kernels. To test this hypothesis, we construct an attention mechanism using randomly initialized orthogonal filters. Integrating this mechanism into ResNet, we create OrthoNet. We compare OrthoNet to FcaNet (and other attention mechanisms) on Birds, MS-COCO, and Places356 and show superior performance. On the ImageNet dataset, our method competes with or surpasses the current state-of-the-art. Our results imply that an optimal choice of filter is elusive and generalization can be achieved with a sufficiently large number of orthogonal filters. We further investigate other general principles for implementing channel attention, such as its position in the network and channel groupings. Our code is publicly available at https://github.com/hady1011/OrthoNets/

Related papers

• Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Tasks [53.974190296524455]
  We revisit ChebNet to shed light on its ability to model distant node interactions.<n>We cast ChebNet as a stable and non-dissipative dynamical system, which we coin Stable-ChebNet.
  arXiv  Detail & Related papers  (2025-06-09T10:41:34Z)
• CFDP: Common Frequency Domain Pruning [0.3021678014343889]
  We introduce a novel end-to-end pipeline for model pruning via the frequency domain. We have achieved state-of-the-art results on CIFAR-10 with GoogLeNet reaching an accuracy of 95.25%, that is, +0.2% from the original model. In addition to notable performances, models produced via CFDP exhibit robustness to a variety of configurations.
  arXiv  Detail & Related papers  (2023-06-07T04:49:26Z)
• GMConv: Modulating Effective Receptive Fields for Convolutional Kernels [52.50351140755224]
  In convolutional neural networks, the convolutions are performed using a square kernel with a fixed N $times$ N receptive field (RF) Inspired by the property that ERFs typically exhibit a Gaussian distribution, we propose a Gaussian Mask convolutional kernel (GMConv) in this work. Our GMConv can directly replace the standard convolutions in existing CNNs and can be easily trained end-to-end by standard back-propagation.
  arXiv  Detail & Related papers  (2023-02-09T10:17:17Z)
• Rega-Net:Retina Gabor Attention for Deep Convolutional Neural Networks [8.068451210598676]
  We propose a novel attention method named Rega-net to increase CNN accuracy by enlarging the receptive field. Inspired by the mechanism of the human retina, we design convolutional kernels to resemble the non-uniformly distributed structure of the human retina.
  arXiv  Detail & Related papers  (2022-11-23T04:24:21Z)
• WaveNets: Wavelet Channel Attention Networks [9.842943431749191]
  We propose Wavelet transform compression as a solution to the channel representation problem. We generalize channel attention using Wavelet compression and name it WaveNet. Our method outperforms the baseline SENet, and achieves the state-of-the-art results.
  arXiv  Detail & Related papers  (2022-11-04T18:26:47Z)
• Graph Neural Networks with Adaptive Frequency Response Filter [55.626174910206046]
  We develop a graph neural network framework AdaGNN with a well-smooth adaptive frequency response filter. We empirically validate the effectiveness of the proposed framework on various benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-26T19:31:21Z)
• FcaNet: Frequency Channel Attention Networks [20.6292655895549]
  We rethink channel attention using frequency analysis. We propose FcaNet with novel multi-spectral channel attention. Our method could improve by 1.8% in terms of Top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2020-12-22T08:33:27Z)
• SCOP: Scientific Control for Reliable Neural Network Pruning [127.20073865874636]
  This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Redundant filters can be discovered in the adversarial process of different features. Our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet.
  arXiv  Detail & Related papers  (2020-10-21T03:02:01Z)
• OrthoReg: Robust Network Pruning Using Orthonormality Regularization [7.754712828900727]
  We propose a principled regularization strategy that enforces orthonormality on a network's filters to reduce inter-filter correlation. When used for iterative pruning on VGG-13, MobileNet-V1, and ResNet-34, OrthoReg consistently outperforms five baseline techniques.
  arXiv  Detail & Related papers  (2020-09-10T17:21:21Z)
• Dependency Aware Filter Pruning [74.69495455411987]
  Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost. Previous work prunes filters according to their weight norms or the corresponding batch-norm scaling factors. We propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.
  arXiv  Detail & Related papers  (2020-05-06T07:41:22Z)
• Filter Grafting for Deep Neural Networks: Reason, Method, and Cultivation [86.91324735966766]
  Filter is the key component in modern convolutional neural networks (CNNs) In this paper, we introduce filter grafting (textbfMethod) to achieve this goal. We develop a novel criterion to measure the information of filters and an adaptive weighting strategy to balance the grafted information among networks.
  arXiv  Detail & Related papers  (2020-04-26T08:36:26Z)
• Filter Grafting for Deep Neural Networks [71.39169475500324]
  Filter grafting aims to improve the representation capability of Deep Neural Networks (DNNs) We develop an entropy-based criterion to measure the information of filters and an adaptive weighting strategy for balancing the grafted information among networks. For example, the grafted MobileNetV2 outperforms the non-grafted MobileNetV2 by about 7 percent on CIFAR-100 dataset.
  arXiv  Detail & Related papers  (2020-01-15T03:18:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.