Generalisation in Neural Networks Does not Require Feature Overlap

• URL: http://arxiv.org/abs/2107.06872v1
• Date: Sun, 4 Jul 2021 09:23:49 GMT
• Title: Generalisation in Neural Networks Does not Require Feature Overlap
• Abstract summary: We show that convolutional architectures avoid the limitation by applying them to two well known challenges. successful performance on the test set requires generalising to features that were not present in the training data. We discuss the role of symmetry in the two tasks and its connection to generalisation.
• Score: 3.162825695928202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: That shared features between train and test data are required for generalisation in artificial neural networks has been a common assumption of both proponents and critics of these models. Here, we show that convolutional architectures avoid this limitation by applying them to two well known challenges, based on learning the identity function and learning rules governing sequences of words. In each case, successful performance on the test set requires generalising to features that were not present in the training data, which is typically not feasible for standard connectionist models. However, our experiments demonstrate that neural networks can succeed on such problems when they incorporate the weight sharing employed by convolutional architectures. In the image processing domain, such architectures are intended to reflect the symmetry under spatial translations of the natural world that such images depict. We discuss the role of symmetry in the two tasks and its connection to generalisation.

Related papers

• Deep Neural Networks as Iterated Function Systems and a Generalization Bound [2.7920304852537536]
  We show that two important deep architectures can be viewed as, or canonically associated with, place-dependent IFS.<n>We derive a Wasserstein bound for generative modeling that controls the collage-type approximation error between the data distribution and its image.
  arXiv  Detail & Related papers  (2026-01-27T07:32:49Z)
• Bearing Syntactic Fruit with Stack-Augmented Neural Networks [59.49467149799849]
  We show that stack-augmented neural networks are more accurate models of human language acquisition than standard architectures.<n>We also propose a modification to the stack RNN architecture that improves hierarchical generalization.
  arXiv  Detail & Related papers  (2025-11-05T15:30:58Z)
• Propositional Logic for Probing Generalization in Neural Networks [3.6037930269014633]
  We investigate the generalization behavior of three key neural architectures (Transformers, Graph Convolution Networks and LSTMs) in a controlled task rooted in propositional logic.<n>We find thatTransformers fail to apply negation compositionally, unless structural biases are introduced.<n>Our findings highlight persistent limitations in the ability of standard architectures to learn systematic representations of logical operators.
  arXiv  Detail & Related papers  (2025-06-10T16:46:05Z)
• Generalization and Estimation Error Bounds for Model-based Neural Networks [78.88759757988761]
  We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks. We derive practical design rules that allow to construct model-based networks with guaranteed high generalization.
  arXiv  Detail & Related papers  (2023-04-19T16:39:44Z)
• Robust Graph Representation Learning via Predictive Coding [46.22695915912123]
  Predictive coding is a message-passing framework initially developed to model information processing in the brain. In this work, we build models that rely on the message-passing rule of predictive coding. We show that the proposed models are comparable to standard ones in terms of performance in both inductive and transductive tasks.
  arXiv  Detail & Related papers  (2022-12-09T03:58:22Z)
• Neural Networks and the Chomsky Hierarchy [27.470857324448136]
  We study whether insights from the theory of Chomsky can predict the limits of neural network generalization in practice. We show negative results where even extensive amounts of data and training time never led to any non-trivial generalization. Our results show that, for our subset of tasks, RNNs and Transformers fail to generalize on non-regular tasks, and only networks augmented with structured memory can successfully generalize on context-free and context-sensitive tasks.
  arXiv  Detail & Related papers  (2022-07-05T15:06:11Z)
• Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation [55.80128181112308]
  We show that dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as network's performance discriminants. Our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces.
  arXiv  Detail & Related papers  (2022-03-30T21:47:32Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Generalization capabilities of translationally equivariant neural networks [0.0]
  In this work, we focus on complex scalar field theory on a two-dimensional lattice and investigate the benefits of using group equivariant convolutional neural network architectures. For a meaningful comparison, we conduct a systematic search for equivariant and non-equivariant neural network architectures and apply them to various regression and classification tasks. We demonstrate that our best equivariant architectures can perform and generalize significantly better than their non-equivariant counterparts.
  arXiv  Detail & Related papers  (2021-03-26T18:53:36Z)
• Seismic horizon detection with neural networks [62.997667081978825]
  This paper is an open-sourced research of applying binary segmentation approach to the task of horizon detection on multiple real seismic cubes with a focus on inter-cube generalization of the predictive model. The main contribution of this paper is an open-sourced research of applying binary segmentation approach to the task of horizon detection on multiple real seismic cubes with a focus on inter-cube generalization of the predictive model.
  arXiv  Detail & Related papers  (2020-01-10T11:30:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.