Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs

• URL: http://arxiv.org/abs/2110.11400v1
• Date: Mon, 18 Oct 2021 22:50:07 GMT
• Title: Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs
• Authors: David Bonet, Antonio Ortega, Javier Ruiz-Hidalgo, Sarath Shekkizhar
• Abstract summary: Feature spaces in the deep layers of convolutional neural networks (CNNs) are often very high-dimensional and difficult to interpret. We analyze theoretically channel-wise non-negative kernel (CW-NNK) regression graphs to quantify the overlap between channels. We find that redundancy between channels is significant and varies with the layer depth and the level of regularization.
• Score: 36.479195100553085
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Feature spaces in the deep layers of convolutional neural networks (CNNs) are often very high-dimensional and difficult to interpret. However, convolutional layers consist of multiple channels that are activated by different types of inputs, which suggests that more insights may be gained by studying the channels and how they relate to each other. In this paper, we first analyze theoretically channel-wise non-negative kernel (CW-NNK) regression graphs, which allow us to quantify the overlap between channels and, indirectly, the intrinsic dimension of the data representation manifold. We find that redundancy between channels is significant and varies with the layer depth and the level of regularization during training. Additionally, we observe that there is a correlation between channel overlap in the last convolutional layer and generalization performance. Our experimental results demonstrate that these techniques can lead to a better understanding of deep representations.

Related papers

• AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space [9.302098067235507]
  We study the intriguing connection between visual data, deep networks, and the brain. Our method creates a universal channel alignment by using brain voxel fMRI response prediction as the training objective.
  arXiv  Detail & Related papers  (2024-06-26T13:38:16Z)
• Joint Channel Estimation and Feedback with Masked Token Transformers in Massive MIMO Systems [74.52117784544758]
  This paper proposes an encoder-decoder based network that unveils the intrinsic frequency-domain correlation within the CSI matrix. The entire encoder-decoder network is utilized for channel compression. Our method outperforms state-of-the-art channel estimation and feedback techniques in joint tasks.
  arXiv  Detail & Related papers  (2023-06-08T06:15:17Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• Low Complexity Channel estimation with Neural Network Solutions [1.0499453838486013]
  We deploy a general residual convolutional neural network to achieve channel estimation in a downlink scenario. Compared with other deep learning methods for channel estimation, our results suggest improved mean squared error computation.
  arXiv  Detail & Related papers  (2022-01-24T19:55:10Z)
• Group Fisher Pruning for Practical Network Compression [58.25776612812883]
  We present a general channel pruning approach that can be applied to various complicated structures. We derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels. Our method can be used to prune any structures including those with coupled channels.
  arXiv  Detail & Related papers  (2021-08-02T08:21:44Z)
• Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation [36.479195100553085]
  Convolutional neural networks (ConvNets) comprise high-dimensional feature spaces formed by the aggregation of multiple channels. We present channel-wise DeepNNK, a novel generalization estimate based on non-dimensional kernel regression (NNK) graphs.
  arXiv  Detail & Related papers  (2021-07-27T17:33:30Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition [126.51241919472356]
  We design a simple and highly modularized graph convolutional network architecture for skeleton-based action recognition. Our network is constructed by repeating a building block that aggregates multi-granularity information from both the spatial and temporal paths.
  arXiv  Detail & Related papers  (2020-11-26T14:43:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.