Contextually Guided Convolutional Neural Networks for Learning Most
Transferable Representations
- URL: http://arxiv.org/abs/2103.01566v1
- Date: Tue, 2 Mar 2021 08:41:12 GMT
- Title: Contextually Guided Convolutional Neural Networks for Learning Most
Transferable Representations
- Authors: Olcay Kursun, Semih Dinc, Oleg V. Favorov
- Abstract summary: We propose an efficient algorithm for developing broad-purpose representations transferable to new tasks without additional training.
A contextually guided CNN (CG-CNN) is trained on groups of neighboring image patches picked at random image locations in the dataset.
In our application to natural images, we find that CG-CNN features show the same, if not higher, transfer utility and classification accuracy as comparable transferable features in the first CNN layer.
- Score: 1.160208922584163
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep Convolutional Neural Networks (CNNs), trained extensively on very large
labeled datasets, learn to recognize inferentially powerful features in their
input patterns and represent efficiently their objective content. Such
objectivity of their internal representations enables deep CNNs to readily
transfer and successfully apply these representations to new classification
tasks. Deep CNNs develop their internal representations through a challenging
process of error backpropagation-based supervised training. In contrast, deep
neural networks of the cerebral cortex develop their even more powerful
internal representations in an unsupervised process, apparently guided at a
local level by contextual information. Implementing such local contextual
guidance principles in a single-layer CNN architecture, we propose an efficient
algorithm for developing broad-purpose representations (i.e., representations
transferable to new tasks without additional training) in shallow CNNs trained
on limited-size datasets. A contextually guided CNN (CG-CNN) is trained on
groups of neighboring image patches picked at random image locations in the
dataset. Such neighboring patches are likely to have a common context and
therefore are treated for the purposes of training as belonging to the same
class. Across multiple iterations of such training on different context-sharing
groups of image patches, CNN features that are optimized in one iteration are
then transferred to the next iteration for further optimization, etc. In this
process, CNN features acquire higher pluripotency, or inferential utility for
any arbitrary classification task, which we quantify as a transfer utility. In
our application to natural images, we find that CG-CNN features show the same,
if not higher, transfer utility and classification accuracy as comparable
transferable features in the first CNN layer of the well-known deep networks.
Related papers
- CNN2GNN: How to Bridge CNN with GNN [59.42117676779735]
We propose a novel CNN2GNN framework to unify CNN and GNN together via distillation.
The performance of distilled boosted'' two-layer GNN on Mini-ImageNet is much higher than CNN containing dozens of layers such as ResNet152.
arXiv Detail & Related papers (2024-04-23T08:19:08Z) - Training Convolutional Neural Networks with the Forward-Forward
algorithm [1.74440662023704]
Forward Forward (FF) algorithm has up to now only been used in fully connected networks.
We show how the FF paradigm can be extended to CNNs.
Our FF-trained CNN, featuring a novel spatially-extended labeling technique, achieves a classification accuracy of 99.16% on the MNIST hand-written digits dataset.
arXiv Detail & Related papers (2023-12-22T18:56:35Z) - A novel feature-scrambling approach reveals the capacity of
convolutional neural networks to learn spatial relations [0.0]
Convolutional neural networks (CNNs) are one of the most successful computer vision systems to solve object recognition.
Yet it remains poorly understood how CNNs actually make their decisions, what the nature of their internal representations is, and how their recognition strategies differ from humans.
arXiv Detail & Related papers (2022-12-12T16:40:29Z) - Knowledge Distillation By Sparse Representation Matching [107.87219371697063]
We propose Sparse Representation Matching (SRM) to transfer intermediate knowledge from one Convolutional Network (CNN) to another by utilizing sparse representation.
We formulate as a neural processing block, which can be efficiently optimized using gradient descent and integrated into any CNN in a plug-and-play manner.
Our experiments demonstrate that is robust to architectural differences between the teacher and student networks, and outperforms other KD techniques across several datasets.
arXiv Detail & Related papers (2021-03-31T11:47:47Z) - The Mind's Eye: Visualizing Class-Agnostic Features of CNNs [92.39082696657874]
We propose an approach to visually interpret CNN features given a set of images by creating corresponding images that depict the most informative features of a specific layer.
Our method uses a dual-objective activation and distance loss, without requiring a generator network nor modifications to the original model.
arXiv Detail & Related papers (2021-01-29T07:46:39Z) - Fusion of CNNs and statistical indicators to improve image
classification [65.51757376525798]
Convolutional Networks have dominated the field of computer vision for the last ten years.
Main strategy to prolong this trend relies on further upscaling networks in size.
We hypothesise that adding heterogeneous sources of information may be more cost-effective to a CNN than building a bigger network.
arXiv Detail & Related papers (2020-12-20T23:24:31Z) - Exploring the Interchangeability of CNN Embedding Spaces [0.5735035463793008]
We map between 10 image-classification CNNs and between 4 facial-recognition CNNs.
For CNNs trained to the same classes and sharing a common backend-logit architecture, a linear-mapping may always be calculated directly from the backend layer weights.
The implications are far-reaching, suggesting an underlying commonality between representations learned by networks designed and trained for a common task.
arXiv Detail & Related papers (2020-10-05T20:32:40Z) - Dynamic Hierarchical Mimicking Towards Consistent Optimization
Objectives [73.15276998621582]
We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability.
Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network.
Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
arXiv Detail & Related papers (2020-03-24T09:56:13Z) - Curriculum By Smoothing [52.08553521577014]
Convolutional Neural Networks (CNNs) have shown impressive performance in computer vision tasks such as image classification, detection, and segmentation.
We propose an elegant curriculum based scheme that smoothes the feature embedding of a CNN using anti-aliasing or low-pass filters.
As the amount of information in the feature maps increases during training, the network is able to progressively learn better representations of the data.
arXiv Detail & Related papers (2020-03-03T07:27:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.