DNNs as Layers of Cooperating Classifiers

• URL: http://arxiv.org/abs/2001.06178v1
• Date: Fri, 17 Jan 2020 07:45:26 GMT
• Title: DNNs as Layers of Cooperating Classifiers
• Authors: Marelie H. Davel, Marthinus W. Theunissen, Arnold M. Pretorius, Etienne Barnard
• Abstract summary: A robust theoretical framework can describe and predict the generalization ability of deep neural networks (DNNs) in general circumstances remains elusive. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration.
• Score: 5.746505534720594
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A robust theoretical framework that can describe and predict the generalization ability of deep neural networks (DNNs) in general circumstances remains elusive. Classical attempts have produced complexity metrics that rely heavily on global measures of compactness and capacity with little investigation into the effects of sub-component collaboration. We demonstrate intriguing regularities in the activation patterns of the hidden nodes within fully-connected feedforward networks. By tracing the origin of these patterns, we show how such networks can be viewed as the combination of two information processing systems: one continuous and one discrete. We describe how these two systems arise naturally from the gradient-based optimization process, and demonstrate the classification ability of the two systems, individually and in collaboration. This perspective on DNN classification offers a novel way to think about generalization, in which different subsets of the training data are used to train distinct classifiers; those classifiers are then combined to perform the classification task, and their consistency is crucial for accurate classification.

Related papers

• Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks [13.983863226803336]
  We argue that "Feature Averaging" is one of the principal factors contributing to non-robustness of deep neural networks. We provide a detailed theoretical analysis of the training dynamics of gradient descent in a two-layer ReLU network for a binary classification task. We prove that, with the provision of more granular supervised information, a two-layer multi-class neural network is capable of learning individual features.
  arXiv  Detail & Related papers  (2024-10-14T09:28:32Z)
• Decomposing neural networks as mappings of correlation functions [57.52754806616669]
  We study the mapping between probability distributions implemented by a deep feed-forward network. We identify essential statistics in the data, as well as different information representations that can be used by neural networks.
  arXiv  Detail & Related papers  (2022-02-10T09:30:31Z)
• ExpertNet: A Symbiosis of Classification and Clustering [22.324813752423044]
  ExpertNet uses novel training strategies to learn clustered latent representations and leverage them by effectively combining cluster-specific classifiers. We demonstrate the superiority of ExpertNet over state-of-the-art methods on 6 large clinical datasets.
  arXiv  Detail & Related papers  (2022-01-17T11:00:30Z)
• Self-Ensembling GAN for Cross-Domain Semantic Segmentation [107.27377745720243]
  This paper proposes a self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain data for semantic segmentation. In SE-GAN, a teacher network and a student network constitute a self-ensembling model for generating semantic segmentation maps, which together with a discriminator, forms a GAN. Despite its simplicity, we find SE-GAN can significantly boost the performance of adversarial training and enhance the stability of the model.
  arXiv  Detail & Related papers  (2021-12-15T09:50:25Z)
• GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition [33.23662792742078]
  We propose a two-stage deep neural network for zero-shot action recognition. In the sampling stage, we utilize a generative adversarial networks (GAN) trained by action features and word vectors of seen classes. In the classification stage, we construct a knowledge graph based on the relationship between word vectors of action classes and related objects.
  arXiv  Detail & Related papers  (2021-05-25T09:34:42Z)
• Intraclass clustering: an implicit learning ability that regularizes DNNs [22.732204569029648]
  We show that deep neural networks are regularized through their ability to extract meaningful clusters among a class. Measures of intraclass clustering are designed based on the neuron- and layer-level representations of the training data.
  arXiv  Detail & Related papers  (2021-03-11T15:26:27Z)
• Binary Classification from Multiple Unlabeled Datasets via Surrogate Set Classification [94.55805516167369]
  We propose a new approach for binary classification from m U-sets for $mge2$. Our key idea is to consider an auxiliary classification task called surrogate set classification (SSC)
  arXiv  Detail & Related papers  (2021-02-01T07:36:38Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Equivalent Classification Mapping for Weakly Supervised Temporal Action Localization [92.58946210982411]
  Weakly supervised temporal action localization is a newly emerging yet widely studied topic in recent years. The pre-classification pipeline first performs classification on each video snippet and then aggregate the snippet-level classification scores to obtain the video-level classification score. The post-classification pipeline aggregates the snippet-level features first and then predicts the video-level classification score based on the aggregated feature.
  arXiv  Detail & Related papers  (2020-08-18T03:54:56Z)
• Fine-Grained Visual Classification with Efficient End-to-end Localization [49.9887676289364]
  We present an efficient localization module that can be fused with a classification network in an end-to-end setup. We evaluate the new model on the three benchmark datasets CUB200-2011, Stanford Cars and FGVC-Aircraft.
  arXiv  Detail & Related papers  (2020-05-11T14:07:06Z)
• A Classification-Based Approach to Semi-Supervised Clustering with Pairwise Constraints [5.639904484784126]
  We introduce a network framework for semi-supervised clustering with pairwise constraints. In contrast to existing approaches, we decompose SSC into two simpler classification tasks/stages. The proposed approach, S3C2, is motivated by the observation that binary classification is usually easier than multi-class clustering.
  arXiv  Detail & Related papers  (2020-01-18T20:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.