Related papers: Coordination Among Neural Modules Through a Shared Global Workspace

Coordination Among Neural Modules Through a Shared Global Workspace

URL: http://arxiv.org/abs/2103.01197v1
Date: Mon, 1 Mar 2021 18:43:48 GMT
Title: Coordination Among Neural Modules Through a Shared Global Workspace
Authors: Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, Yoshua Bengio
Abstract summary: In cognitive science, a global workspace architecture has been proposed in which functionally specialized components share information. We show that capacity limitations have a rational basis in that they encourage specialization and compositionality.
Score: 78.08062292790109
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Deep learning has seen a movement away from representing examples with a monolithic hidden state towards a richly structured state. For example, Transformers segment by position, and object-centric architectures decompose images into entities. In all these architectures, interactions between different elements are modeled via pairwise interactions: Transformers make use of self-attention to incorporate information from other positions; object-centric architectures make use of graph neural networks to model interactions among entities. However, pairwise interactions may not achieve global coordination or a coherent, integrated representation that can be used for downstream tasks. In cognitive science, a global workspace architecture has been proposed in which functionally specialized components share information through a common, bandwidth-limited communication channel. We explore the use of such a communication channel in the context of deep learning for modeling the structure of complex environments. The proposed method includes a shared workspace through which communication among different specialist modules takes place but due to limits on the communication bandwidth, specialist modules must compete for access. We show that capacity limitations have a rational basis in that (1) they encourage specialization and compositionality and (2) they facilitate the synchronization of otherwise independent specialists.