Compositional Languages Emerge in a Neural Iterated Learning Model

• URL: http://arxiv.org/abs/2002.01365v2
• Date: Mon, 17 Feb 2020 11:22:04 GMT
• Title: Compositional Languages Emerge in a Neural Iterated Learning Model
• Authors: Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, Simon Kirby
• Abstract summary: compositionality enables natural language to represent complex concepts via a structured combination of simpler ones. We propose an effective neural iterated learning (NIL) algorithm that, when applied to interacting neural agents, facilitates the emergence of a more structured type of language.
• Score: 27.495624644227888
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The principle of compositionality, which enables natural language to represent complex concepts via a structured combination of simpler ones, allows us to convey an open-ended set of messages using a limited vocabulary. If compositionality is indeed a natural property of language, we may expect it to appear in communication protocols that are created by neural agents in language games. In this paper, we propose an effective neural iterated learning (NIL) algorithm that, when applied to interacting neural agents, facilitates the emergence of a more structured type of language. Indeed, these languages provide learning speed advantages to neural agents during training, which can be incrementally amplified via NIL. We provide a probabilistic model of NIL and an explanation of why the advantage of compositional language exist. Our experiments confirm our analysis, and also demonstrate that the emerged languages largely improve the generalizing power of the neural agent communication.

Related papers

• SNeL: A Structured Neuro-Symbolic Language for Entity-Based Multimodal Scene Understanding [0.0]
  We introduce SNeL (Structured Neuro-symbolic Language), a versatile query language designed to facilitate nuanced interactions with neural networks processing multimodal data. SNeL's expressive interface enables the construction of intricate queries, supporting logical and arithmetic operators, comparators, nesting, and more. Our evaluations demonstrate SNeL's potential to reshape the way we interact with complex neural networks.
  arXiv  Detail & Related papers  (2023-06-09T17:01:51Z)
• On the Correspondence between Compositionality and Imitation in Emergent Neural Communication [1.4610038284393165]
  Our work explores the link between compositionality and imitation in a Lewis game played by deep neural agents. supervised learning tends to produce more average languages, while reinforcement learning introduces a selection pressure toward more compositional languages.
  arXiv  Detail & Related papers  (2023-05-22T11:41:29Z)
• What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure [5.871583927216651]
  A fundamental property of language is its compositional structure, allowing humans to produce forms for new meanings. For humans, languages with more compositional and transparent structures are typically easier to learn than those with opaque and irregular structures. This learnability advantage has not yet been shown for deep neural networks, limiting their use as models for human language learning.
  arXiv  Detail & Related papers  (2023-02-23T18:57:34Z)
• Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-off [3.631024220680066]
  We propose a new Neural-agent Language Learning and Communication framework (NeLLCom) where pairs of speaking and listening agents first learn a miniature language. We succeed in replicating the trade-off with the new framework without hard-coding specific biases in the agents.
  arXiv  Detail & Related papers  (2023-01-30T17:22:33Z)
• Benchmarking Compositionality with Formal Languages [64.09083307778951]
  We investigate whether large neural models in NLP can acquire the ability tocombining primitive concepts into larger novel combinations while learning from data. By randomly sampling over many transducers, we explore which of their properties contribute to learnability of a compositional relation by a neural network. We find that the models either learn the relations completely or not at all. The key is transition coverage, setting a soft learnability limit at 400 examples per transition.
  arXiv  Detail & Related papers  (2022-08-17T10:03:18Z)
• Linking Emergent and Natural Languages via Corpus Transfer [98.98724497178247]
  We propose a novel way to establish a link by corpus transfer between emergent languages and natural languages. Our approach showcases non-trivial transfer benefits for two different tasks -- language modeling and image captioning. We also introduce a novel metric to predict the transferability of an emergent language by translating emergent messages to natural language captions grounded on the same images.
  arXiv  Detail & Related papers  (2022-03-24T21:24:54Z)
• Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models [32.27333420000134]
  We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language.
  arXiv  Detail & Related papers  (2022-03-19T13:29:48Z)
• Dependency-based Mixture Language Models [53.152011258252315]
  We introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
  arXiv  Detail & Related papers  (2022-03-19T06:28:30Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Compositional Processing Emerges in Neural Networks Solving Math Problems [100.80518350845668]
  Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed. Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
  arXiv  Detail & Related papers  (2021-05-19T07:24:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.