Recurrent neural networks that generalize from examples and optimize by dreaming

• URL: http://arxiv.org/abs/2204.07954v1
• Date: Sun, 17 Apr 2022 08:40:54 GMT
• Title: Recurrent neural networks that generalize from examples and optimize by dreaming
• Authors: Miriam Aquaro, Francesco Alemanno, Ido Kanter, Fabrizio Durante, Elena Agliari, Adriano Barra
• Abstract summary: We introduce a generalized Hopfield network where pairwise couplings between neurons are built according to Hebb's prescription for on-line learning. We let the network experience solely a dataset made of a sample of noisy examples for each pattern. Remarkably, the sleeping mechanisms always significantly reduce the dataset size required to correctly generalize.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The gap between the huge volumes of data needed to train artificial neural networks and the relatively small amount of data needed by their biological counterparts is a central puzzle in machine learning. Here, inspired by biological information-processing, we introduce a generalized Hopfield network where pairwise couplings between neurons are built according to Hebb's prescription for on-line learning and allow also for (suitably stylized) off-line sleeping mechanisms. Moreover, in order to retain a learning framework, here the patterns are not assumed to be available, instead, we let the network experience solely a dataset made of a sample of noisy examples for each pattern. We analyze the model by statistical-mechanics tools and we obtain a quantitative picture of its capabilities as functions of its control parameters: the resulting network is an associative memory for pattern recognition that learns from examples on-line, generalizes and optimizes its storage capacity by off-line sleeping. Remarkably, the sleeping mechanisms always significantly reduce (up to $\approx 90\%$) the dataset size required to correctly generalize, further, there are memory loads that are prohibitive to Hebbian networks without sleeping (no matter the size and quality of the provided examples), but that are easily handled by the present "rested" neural networks.

Related papers

• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Spiking representation learning for associative memories [0.0]
  We introduce a novel artificial spiking neural network (SNN) that performs unsupervised representation learning and associative memory operations. The architecture of our model derives from the neocortical columnar organization and combines feedforward projections for learning hidden representations and recurrent projections for forming associative memories.
  arXiv  Detail & Related papers  (2024-06-05T08:30:11Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Heterogenous Memory Augmented Neural Networks [84.29338268789684]
  We introduce a novel heterogeneous memory augmentation approach for neural networks. By introducing learnable memory tokens with attention mechanism, we can effectively boost performance without huge computational overhead. We show our approach on various image and graph-based tasks under both in-distribution (ID) and out-of-distribution (OOD) conditions.
  arXiv  Detail & Related papers  (2023-10-17T01:05:28Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Robust Generalization of Quadratic Neural Networks via Function Identification [19.87036824512198]
  Generalization bounds from learning theory often assume that the test distribution is close to the training distribution. We show that for quadratic neural networks, we can identify the function represented by the model even though we cannot identify its parameters.
  arXiv  Detail & Related papers  (2021-09-22T18:02:00Z)
• Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry [55.57953219617467]
  We show that the learnability of a neural network is directly related to its size. We use Betti numbers to measure the topological geometric complexity of input data and the neural network. We perform the experiments on a real-world dataset MNIST and the results verify our analysis and conclusion.
  arXiv  Detail & Related papers  (2021-09-03T11:45:51Z)
• Slope and generalization properties of neural networks [0.0]
  We show that the distribution of the slope of a well-trained neural network classifier is generally independent of the width of the layers in a fully connected network. The slope is of similar size throughout the relevant volume, and varies smoothly. It also behaves as predicted in rescaling examples. We discuss possible applications of the slope concept, such as using it as a part of the loss function or stopping criterion during network training, or ranking data sets in terms of their complexity.
  arXiv  Detail & Related papers  (2021-07-03T17:54:27Z)
• Locally Sparse Networks for Interpretable Predictions [7.362415721170984]
  We propose a framework for training locally sparse neural networks where the local sparsity is learned via a sample-specific gating mechanism. The sample-specific sparsity is predicted via a textitgating network, which is trained in tandem with the textitprediction network. We demonstrate that our method outperforms state-of-the-art models when predicting the target function with far fewer features per instance.
  arXiv  Detail & Related papers  (2021-06-11T15:46:50Z)
• Information contraction in noisy binary neural networks and its implications [11.742803725197506]
  We consider noisy binary neural networks, where each neuron has a non-zero probability of producing an incorrect output. Our key finding is a lower bound for the required number of neurons in noisy neural networks, which is first of its kind. This paper offers new understanding of noisy information processing systems through the lens of information theory.
  arXiv  Detail & Related papers  (2021-01-28T00:01:45Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.