The Relational Bottleneck as an Inductive Bias for Efficient Abstraction

• URL: http://arxiv.org/abs/2309.06629v5
• Date: Wed, 1 May 2024 05:36:30 GMT
• Title: The Relational Bottleneck as an Inductive Bias for Efficient Abstraction
• Authors: Taylor W. Webb, Steven M. Frankland, Awni Altabaa, Simon Segert, Kamesh Krishnamurthy, Declan Campbell, Jacob Russin, Tyler Giallanza, Zack Dulberg, Randall O'Reilly, John Lafferty, Jonathan D. Cohen,
• Abstract summary: We show that neural networks are constrained via their architecture to focus on relations between perceptual inputs, rather than the attributes of individual inputs. We review a family of models that employ this approach to induce abstractions in a data-efficient manner.
• Score: 3.19883356005403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A central challenge for cognitive science is to explain how abstract concepts are acquired from limited experience. This has often been framed in terms of a dichotomy between connectionist and symbolic cognitive models. Here, we highlight a recently emerging line of work that suggests a novel reconciliation of these approaches, by exploiting an inductive bias that we term the relational bottleneck. In that approach, neural networks are constrained via their architecture to focus on relations between perceptual inputs, rather than the attributes of individual inputs. We review a family of models that employ this approach to induce abstractions in a data-efficient manner, emphasizing their potential as candidate models for the acquisition of abstract concepts in the human mind and brain.

Related papers

• Abstraction Alignment: Comparing Model and Human Conceptual Relationships [26.503178592074757]
  We introduce abstraction alignment, a methodology to measure the agreement between a model's learned abstraction and the expected human abstraction. In evaluation tasks, abstraction alignment provides a deeper understanding of model behavior and dataset content.
  arXiv  Detail & Related papers  (2024-07-17T13:27:26Z)
• Automatic Discovery of Visual Circuits [66.99553804855931]
  We explore scalable methods for extracting the subgraph of a vision model's computational graph that underlies recognition of a specific visual concept. We find that our approach extracts circuits that causally affect model output, and that editing these circuits can defend large pretrained models from adversarial attacks.
  arXiv  Detail & Related papers  (2024-04-22T17:00:57Z)
• A Relational Inductive Bias for Dimensional Abstraction in Neural Networks [3.5063551678446494]
  This paper investigates the impact of the relational bottleneck on the learning of factorized representations conducive to compositional coding. We demonstrate that such a bottleneck not only improves generalization and learning efficiency, but also aligns network performance with human-like behavioral biases.
  arXiv  Detail & Related papers  (2024-02-28T15:51:05Z)
• Neural Causal Abstractions [63.21695740637627]
  We develop a new family of causal abstractions by clustering variables and their domains. We show that such abstractions are learnable in practical settings through Neural Causal Models. Our experiments support the theory and illustrate how to scale causal inferences to high-dimensional settings involving image data.
  arXiv  Detail & Related papers  (2024-01-05T02:00:27Z)
• A Cognitively-Inspired Neural Architecture for Visual Abstract Reasoning Using Contrastive Perceptual and Conceptual Processing [14.201935774784632]
  We introduce a new neural architecture for solving visual abstract reasoning tasks inspired by human cognition. Inspired by this principle, our architecture models visual abstract reasoning as an iterative, self-contrasting learning process. Experiments on the machine learning dataset RAVEN show that CPCNet achieves higher accuracy than all previously published models.
  arXiv  Detail & Related papers  (2023-09-19T11:18:01Z)
• On Neural Architecture Inductive Biases for Relational Tasks [76.18938462270503]
  We introduce a simple architecture based on similarity-distribution scores which we name Compositional Network generalization (CoRelNet) We find that simple architectural choices can outperform existing models in out-of-distribution generalizations.
  arXiv  Detail & Related papers  (2022-06-09T16:24:01Z)
• Acquiring and Modelling Abstract Commonsense Knowledge via Conceptualization [49.00409552570441]
  We study the role of conceptualization in commonsense reasoning, and formulate a framework to replicate human conceptual induction. We apply the framework to ATOMIC, a large-scale human-annotated CKG, aided by the taxonomy Probase.
  arXiv  Detail & Related papers  (2022-06-03T12:24:49Z)
• Proactive Pseudo-Intervention: Causally Informed Contrastive Learning For Interpretable Vision Models [103.64435911083432]
  We present a novel contrastive learning strategy called it Proactive Pseudo-Intervention (PPI) PPI leverages proactive interventions to guard against image features with no causal relevance. We also devise a novel causally informed salience mapping module to identify key image pixels to intervene, and show it greatly facilitates model interpretability.
  arXiv  Detail & Related papers  (2020-12-06T20:30:26Z)
• Developing Constrained Neural Units Over Time [81.19349325749037]
  This paper focuses on an alternative way of defining Neural Networks, that is different from the majority of existing approaches. The structure of the neural architecture is defined by means of a special class of constraints that are extended also to the interaction with data. The proposed theory is cast into the time domain, in which data are presented to the network in an ordered manner.
  arXiv  Detail & Related papers  (2020-09-01T09:07:25Z)
• Learning Representations that Support Extrapolation [39.84463809100903]
  We consider the challenge of learning representations that support extrapolation. We introduce a novel visual analogy benchmark that allows the graded evaluation of extrapolation. We also introduce a simple technique, temporal context normalization, that encourages representations that emphasize the relations between objects.
  arXiv  Detail & Related papers  (2020-07-09T20:53:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.