Adaptive Hierarchical Decomposition of Large Deep Networks
- URL: http://arxiv.org/abs/2008.00809v1
- Date: Fri, 17 Jul 2020 21:04:50 GMT
- Title: Adaptive Hierarchical Decomposition of Large Deep Networks
- Authors: Sumanth Chennupati, Sai Nooka, Shagan Sah, Raymond W Ptucha
- Abstract summary: As datasets get larger, a natural question is if existing deep learning architectures can be extended to handle the 50+K classes thought to be perceptible by a typical human.
This paper introduces a framework that automatically analyzes and configures a family of smaller deep networks as a replacement to a singular, larger network.
The resulting smaller networks are highly scalable, parallel and more practical to train, and achieve higher classification accuracy.
- Score: 4.272649614101117
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep learning has recently demonstrated its ability to rival the human brain
for visual object recognition. As datasets get larger, a natural question to
ask is if existing deep learning architectures can be extended to handle the
50+K classes thought to be perceptible by a typical human. Most deep learning
architectures concentrate on splitting diverse categories, while ignoring the
similarities amongst them. This paper introduces a framework that automatically
analyzes and configures a family of smaller deep networks as a replacement to a
singular, larger network. Class similarities guide the creation of a family
from course to fine classifiers which solve categorical problems more
effectively than a single large classifier. The resulting smaller networks are
highly scalable, parallel and more practical to train, and achieve higher
classification accuracy. This paper also proposes a method to adaptively select
the configuration of the hierarchical family of classifiers using linkage
statistics from overall and sub-classification confusion matrices. Depending on
the number of classes and the complexity of the problem, a deep learning model
is selected and the complexity is determined. Numerous experiments on network
classes, layers, and architecture configurations validate our results.
Related papers
- Informed deep hierarchical classification: a non-standard analysis inspired approach [0.0]
It consists in a multi-output deep neural network equipped with specific projection operators placed before each output layer.
The design of such an architecture, called lexicographic hybrid deep neural network (LH-DNN), has been possible by combining tools from different and quite distant research fields.
To assess the efficacy of the approach, the resulting network is compared against the B-CNN, a convolutional neural network tailored for hierarchical classification tasks.
arXiv Detail & Related papers (2024-09-25T14:12:50Z) - How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model [47.617093812158366]
We introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images.
We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups.
Our results indicate how deep networks overcome the curse of dimensionality by building invariant representations.
arXiv Detail & Related papers (2023-07-05T09:11:09Z) - Dynamic Perceiver for Efficient Visual Recognition [87.08210214417309]
We propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task.
A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks.
Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features.
arXiv Detail & Related papers (2023-06-20T03:00:22Z) - Do We Really Need a Learnable Classifier at the End of Deep Neural
Network? [118.18554882199676]
We study the potential of learning a neural network for classification with the classifier randomly as an ETF and fixed during training.
Our experimental results show that our method is able to achieve similar performances on image classification for balanced datasets.
arXiv Detail & Related papers (2022-03-17T04:34:28Z) - Connecting Sphere Manifolds Hierarchically for Regularization [16.082095595061617]
We consider classification problems with hierarchically organized classes.
Our technique replaces the last layer of a neural network by combining a spherical fully-connected layer with a hierarchical layer.
This regularization is shown to improve the performance of widely used deep neural network architectures.
arXiv Detail & Related papers (2021-06-25T10:51:36Z) - Firefly Neural Architecture Descent: a General Approach for Growing
Neural Networks [50.684661759340145]
Firefly neural architecture descent is a general framework for progressively and dynamically growing neural networks.
We show that firefly descent can flexibly grow networks both wider and deeper, and can be applied to learn accurate but resource-efficient neural architectures.
In particular, it learns networks that are smaller in size but have higher average accuracy than those learned by the state-of-the-art methods.
arXiv Detail & Related papers (2021-02-17T04:47:18Z) - Gradient-based Competitive Learning: Theory [1.6752712949948443]
This paper introduces a novel perspective in this area by combining gradient-based and competitive learning.
The theory is based on the intuition that neural networks are able to learn topological structures by working directly on the transpose of the input matrix.
The proposed approach has a great potential as it can be generalized to a vast selection of topological learning tasks.
arXiv Detail & Related papers (2020-09-06T19:00:51Z) - Automated Search for Resource-Efficient Branched Multi-Task Networks [81.48051635183916]
We propose a principled approach, rooted in differentiable neural architecture search, to automatically define branching structures in a multi-task neural network.
We show that our approach consistently finds high-performing branching structures within limited resource budgets.
arXiv Detail & Related papers (2020-08-24T09:49:19Z) - DC-NAS: Divide-and-Conquer Neural Architecture Search [108.57785531758076]
We present a divide-and-conquer (DC) approach to effectively and efficiently search deep neural architectures.
We achieve a $75.1%$ top-1 accuracy on the ImageNet dataset, which is higher than that of state-of-the-art methods using the same search space.
arXiv Detail & Related papers (2020-05-29T09:02:16Z) - Learn Class Hierarchy using Convolutional Neural Networks [0.9569316316728905]
We propose a new architecture for hierarchical classification of images, introducing a stack of deep linear layers with cross-entropy loss functions and center loss combined.
We experimentally show that our hierarchical classifier presents advantages to the traditional classification approaches finding application in computer vision tasks.
arXiv Detail & Related papers (2020-05-18T12:06:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.