Feature-Dependent Cross-Connections in Multi-Path Neural Networks
- URL: http://arxiv.org/abs/2006.13904v2
- Date: Fri, 1 Jan 2021 16:49:47 GMT
- Title: Feature-Dependent Cross-Connections in Multi-Path Neural Networks
- Authors: Dumindu Tissera, Kasun Vithanage, Rukshan Wijesinghe, Kumara
Kahatapitiya, Subha Fernando, Ranga Rodrigo
- Abstract summary: Multi-path networks tend to learn redundant features.
We introduce a mechanism to intelligently allocate incoming feature maps to such paths.
We show improved image recognition accuracy at a similar complexity compared to conventional and state-of-the-art methods.
- Score: 7.230526683545722
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning a particular task from a dataset, samples in which originate from
diverse contexts, is challenging, and usually addressed by deepening or
widening standard neural networks. As opposed to conventional network widening,
multi-path architectures restrict the quadratic increment of complexity to a
linear scale. However, existing multi-column/path networks or model ensembling
methods do not consider any feature-dependent allocation of parallel resources,
and therefore, tend to learn redundant features. Given a layer in a multi-path
network, if we restrict each path to learn a context-specific set of features
and introduce a mechanism to intelligently allocate incoming feature maps to
such paths, each path can specialize in a certain context, reducing the
redundancy and improving the quality of extracted features. This eventually
leads to better-optimized usage of parallel resources. To do this, we propose
inserting feature-dependent cross-connections between parallel sets of feature
maps in successive layers. The weighting coefficients of these
cross-connections are computed from the input features of the particular layer.
Our multi-path networks show improved image recognition accuracy at a similar
complexity compared to conventional and state-of-the-art methods for deepening,
widening and adaptive feature extracting, in both small and large scale
datasets.
Related papers
- GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning [51.677086019209554]
We propose a Generalized Structural Sparse to capture powerful relationships across modalities for pair-wise similarity learning.
The distance metric delicately encapsulates two formats of diagonal and block-diagonal terms.
Experiments on cross-modal and two extra uni-modal retrieval tasks have validated its superiority and flexibility.
arXiv Detail & Related papers (2024-10-20T03:45:50Z) - SDA-$x$Net: Selective Depth Attention Networks for Adaptive Multi-scale
Feature Representation [14.7929472540577]
Existing multi-scale solutions lead to a risk of just increasing the receptive field sizes while neglecting small receptive fields.
We introduce a new attention dimension, i.e. depth, in addition to existing attention dimensions such as channel, spatial, and branch.
We present a novel selective depth attention network to symmetrically handle multi-scale objects in various vision tasks.
arXiv Detail & Related papers (2022-09-21T12:49:55Z) - SIRe-Networks: Skip Connections over Interlaced Multi-Task Learning and
Residual Connections for Structure Preserving Object Classification [28.02302915971059]
In this paper, we introduce an interlaced multi-task learning strategy, defined SIRe, to reduce the vanishing gradient in relation to the object classification task.
The presented methodology directly improves a convolutional neural network (CNN) by enforcing the input image structure preservation through auto-encoders.
To validate the presented methodology, a simple CNN and various implementations of famous networks are extended via the SIRe strategy and extensively tested on the CIFAR100 dataset.
arXiv Detail & Related papers (2021-10-06T13:54:49Z) - Multi-scale Matching Networks for Semantic Correspondence [38.904735120815346]
The proposed method achieves state-of-the-art performance on three popular benchmarks with high computational efficiency.
Our multi-scale matching network can be trained end-to-end easily with few additional learnable parameters.
arXiv Detail & Related papers (2021-07-31T10:57:24Z) - End-To-End Data-Dependent Routing in Multi-Path Neural Networks [0.9507070656654633]
We propose the use of multi-path neural networks with data-dependent resource allocation among parallel computations within layers.
Our networks show superior performance to existing widening and adaptive feature extraction, and even ensembles, and deeper networks at similar complexity in the image recognition task.
arXiv Detail & Related papers (2021-07-06T07:58:07Z) - All at Once Network Quantization via Collaborative Knowledge Transfer [56.95849086170461]
We develop a novel collaborative knowledge transfer approach for efficiently training the all-at-once quantization network.
Specifically, we propose an adaptive selection strategy to choose a high-precision enquoteteacher for transferring knowledge to the low-precision student.
To effectively transfer knowledge, we develop a dynamic block swapping method by randomly replacing the blocks in the lower-precision student network with the corresponding blocks in the higher-precision teacher network.
arXiv Detail & Related papers (2021-03-02T03:09:03Z) - Automated Search for Resource-Efficient Branched Multi-Task Networks [81.48051635183916]
We propose a principled approach, rooted in differentiable neural architecture search, to automatically define branching structures in a multi-task neural network.
We show that our approach consistently finds high-performing branching structures within limited resource budgets.
arXiv Detail & Related papers (2020-08-24T09:49:19Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z) - Recursive Multi-model Complementary Deep Fusion forRobust Salient Object
Detection via Parallel Sub Networks [62.26677215668959]
Fully convolutional networks have shown outstanding performance in the salient object detection (SOD) field.
This paper proposes a wider'' network architecture which consists of parallel sub networks with totally different network architectures.
Experiments on several famous benchmarks clearly demonstrate the superior performance, good generalization, and powerful learning ability of the proposed wider framework.
arXiv Detail & Related papers (2020-08-07T10:39:11Z) - Sequential Hierarchical Learning with Distribution Transformation for
Image Super-Resolution [83.70890515772456]
We build a sequential hierarchical learning super-resolution network (SHSR) for effective image SR.
We consider the inter-scale correlations of features, and devise a sequential multi-scale block (SMB) to progressively explore the hierarchical information.
Experiment results show SHSR achieves superior quantitative performance and visual quality to state-of-the-art methods.
arXiv Detail & Related papers (2020-07-19T01:35:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.