How Much Position Information Do Convolutional Neural Networks Encode?
- URL: http://arxiv.org/abs/2001.08248v1
- Date: Wed, 22 Jan 2020 19:44:43 GMT
- Title: How Much Position Information Do Convolutional Neural Networks Encode?
- Authors: Md Amirul Islam, Sen Jia, Neil D. B. Bruce
- Abstract summary: In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent.
In this paper, we test this hypothesis revealing the surprising degree of absolute position information that is encoded in commonly used neural networks.
- Score: 27.604154992915863
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In contrast to fully connected networks, Convolutional Neural Networks (CNNs)
achieve efficiency by learning weights associated with local filters with a
finite spatial extent. An implication of this is that a filter may know what it
is looking at, but not where it is positioned in the image. Information
concerning absolute position is inherently useful, and it is reasonable to
assume that deep CNNs may implicitly learn to encode this information if there
is a means to do so. In this paper, we test this hypothesis revealing the
surprising degree of absolute position information that is encoded in commonly
used neural networks. A comprehensive set of experiments show the validity of
this hypothesis and shed light on how and where this information is represented
while offering clues to where positional information is derived from in deep
CNNs.
Related papers
- What Can Be Learnt With Wide Convolutional Neural Networks? [69.55323565255631]
We study infinitely-wide deep CNNs in the kernel regime.
We prove that deep CNNs adapt to the spatial scale of the target function.
We conclude by computing the generalisation error of a deep CNN trained on the output of another deep CNN.
arXiv Detail & Related papers (2022-08-01T17:19:32Z) - CondenseNeXt: An Ultra-Efficient Deep Neural Network for Embedded
Systems [0.0]
A Convolutional Neural Network (CNN) is a class of Deep Neural Network (DNN) widely used in the analysis of visual images captured by an image sensor.
In this paper, we propose a neoteric variant of deep convolutional neural network architecture to ameliorate the performance of existing CNN architectures for real-time inference on embedded systems.
arXiv Detail & Related papers (2021-12-01T18:20:52Z) - Global Pooling, More than Meets the Eye: Position Information is Encoded
Channel-Wise in CNNs [32.81128493853064]
We demonstrate that positional information is encoded based on the ordering of the channel dimensions, while semantic information is largely not.
We show the real world impact of these findings by applying them to two applications.
arXiv Detail & Related papers (2021-08-17T21:27:30Z) - Reasoning-Modulated Representations [85.08205744191078]
We study a common setting where our task is not purely opaque.
Our approach paves the way for a new class of data-efficient representation learning.
arXiv Detail & Related papers (2021-07-19T13:57:13Z) - Learning Structures for Deep Neural Networks [99.8331363309895]
We propose to adopt the efficient coding principle, rooted in information theory and developed in computational neuroscience.
We show that sparse coding can effectively maximize the entropy of the output signals.
Our experiments on a public image classification dataset demonstrate that using the structure learned from scratch by our proposed algorithm, one can achieve a classification accuracy comparable to the best expert-designed structure.
arXiv Detail & Related papers (2021-05-27T12:27:24Z) - A neural anisotropic view of underspecification in deep learning [60.119023683371736]
We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation.
Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
arXiv Detail & Related papers (2021-04-29T14:31:09Z) - The Connection Between Approximation, Depth Separation and Learnability
in Neural Networks [70.55686685872008]
We study the connection between learnability and approximation capacity.
We show that learnability with deep networks of a target function depends on the ability of simpler classes to approximate the target.
arXiv Detail & Related papers (2021-01-31T11:32:30Z) - Position, Padding and Predictions: A Deeper Look at Position Information
in CNNs [30.583407443282365]
We show that a surprising degree of absolute position information is encoded in commonly used CNNs.
We show that zero padding drives CNNs to encode position information in their internal representations, while a lack of padding precludes position encoding.
This gives rise to deeper questions about the role of position information in CNNs.
arXiv Detail & Related papers (2021-01-28T23:40:32Z) - Shape or Texture: Understanding Discriminative Features in CNNs [28.513300496205044]
Recent studies have shown that CNNs actually exhibit a texture bias'
We show that a network learns the majority of overall shape information at the first few epochs of training.
We also show that the encoding of shape does not imply the encoding of localized per-pixel semantic information.
arXiv Detail & Related papers (2021-01-27T18:54:00Z) - Ventral-Dorsal Neural Networks: Object Detection via Selective Attention [51.79577908317031]
We propose a new framework called Ventral-Dorsal Networks (VDNets)
Inspired by the structure of the human visual system, we propose the integration of a "Ventral Network" and a "Dorsal Network"
Our experimental results reveal that the proposed method outperforms state-of-the-art object detection approaches.
arXiv Detail & Related papers (2020-05-15T23:57:36Z) - An Information-theoretic Visual Analysis Framework for Convolutional
Neural Networks [11.15523311079383]
We introduce a data model to organize the data that can be extracted from CNN models.
We then propose two ways to calculate entropy under different circumstances.
We develop a visual analysis system, CNNSlicer, to interactively explore the amount of information changes inside the model.
arXiv Detail & Related papers (2020-05-02T21:36:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.