Neural networks adapting to datasets: learning network size and topology

• URL: http://arxiv.org/abs/2006.12195v2
• Date: Wed, 15 Jul 2020 10:00:07 GMT
• Title: Neural networks adapting to datasets: learning network size and topology
• Authors: Romuald A. Janik, Aleksandra Nowak
• Abstract summary: We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset.
• Score: 77.34726150561087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a standard gradient-based training. The resulting network has the structure of a graph tailored to the particular learning task and dataset. The obtained networks can also be trained from scratch and achieve virtually identical performance. We explore the properties of the network architectures for a number of datasets of varying difficulty observing systematic regularities. The obtained graphs can be therefore understood as encoding nontrivial characteristics of the particular classification tasks.

Related papers

• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Topological Uncertainty: Monitoring trained neural networks through persistence of activation graphs [0.9786690381850356]
  In industrial applications, data coming from an open-world setting might widely differ from the benchmark datasets on which a network was trained. We develop a method to monitor trained neural networks based on the topological properties of their activation graphs.
  arXiv  Detail & Related papers  (2021-05-07T14:16:03Z)
• Graph-Based Neural Network Models with Multiple Self-Supervised Auxiliary Tasks [79.28094304325116]
  Graph Convolutional Networks are among the most promising approaches for capturing relationships among structured data points. We propose three novel self-supervised auxiliary tasks to train graph-based neural network models in a multi-task fashion.
  arXiv  Detail & Related papers  (2020-11-14T11:09:51Z)
• A Topological Framework for Deep Learning [0.7310043452300736]
  We show that the classification problem in machine learning is always solvable under very mild conditions. In particular, we show that a softmax classification network acts on an input topological space by a finite sequence of topological moves to achieve the classification task.
  arXiv  Detail & Related papers  (2020-08-31T15:56:42Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• An analytic theory of shallow networks dynamics for hinge loss classification [14.323962459195771]
  We study the training dynamics of a simple type of neural network: a single hidden layer trained to perform a classification task. We specialize our theory to the prototypical case of a linearly separable dataset and a linear hinge loss. This allow us to address in a simple setting several phenomena appearing in modern networks such as slowing down of training dynamics, crossover between rich and lazy learning, and overfitting.
  arXiv  Detail & Related papers  (2020-06-19T16:25:29Z)
• GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training [62.73470368851127]
  Graph representation learning has emerged as a powerful technique for addressing real-world problems. We design Graph Contrastive Coding -- a self-supervised graph neural network pre-training framework. We conduct experiments on three graph learning tasks and ten graph datasets.
  arXiv  Detail & Related papers  (2020-06-17T16:18:35Z)
• Progressive Graph Convolutional Networks for Semi-Supervised Node Classification [97.14064057840089]
  Graph convolutional networks have been successful in addressing graph-based tasks such as semi-supervised node classification. We propose a method to automatically build compact and task-specific graph convolutional networks.
  arXiv  Detail & Related papers  (2020-03-27T08:32:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.