Emergent communication and learning pressures in language models: a language evolution perspective
- URL: http://arxiv.org/abs/2403.14427v1
- Date: Thu, 21 Mar 2024 14:33:34 GMT
- Title: Emergent communication and learning pressures in language models: a language evolution perspective
- Authors: Lukas Galke, Limor Raviv,
- Abstract summary: We find that the emergent communication literature excels at designing and adapting models to recover initially absent linguistic phenomena of natural languages.
We identify key pressures that have recovered initially absent human patterns in emergent communication models.
This may serve as inspiration for how to design language models for language acquisition and language evolution research.
- Score: 5.371337604556311
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Language models and humans are two types of learning systems. Finding or facilitating commonalities could enable major breakthroughs in our understanding of the acquisition and evolution of language. Many theories of language evolution rely heavily on learning biases and learning pressures. Yet due to substantial differences in learning pressures, it is questionable whether the similarity between humans and machines is sufficient for insights to carry over and to be worth testing with human participants. Here, we review the emergent communication literature, a subfield of multi-agent reinforcement learning, from a language evolution perspective. We find that the emergent communication literature excels at designing and adapting models to recover initially absent linguistic phenomena of natural languages. Based on a short literature review, we identify key pressures that have recovered initially absent human patterns in emergent communication models: communicative success, efficiency, learnability, and other psycho-/sociolinguistic factors. We argue that this may serve as inspiration for how to design language models for language acquisition and language evolution research.
Related papers
- Language Evolution with Deep Learning [49.879239655532324]
Computational modeling plays an essential role in the study of language emergence.
It aims to simulate the conditions and learning processes that could trigger the emergence of a structured language.
This chapter explores another class of computational models that have recently revolutionized the field of machine learning: deep learning models.
arXiv Detail & Related papers (2024-03-18T16:52:54Z) - Towards More Human-like AI Communication: A Review of Emergent
Communication Research [0.0]
Emergent communication (Emecom) is a field of research aiming to develop artificial agents capable of using natural language.
In this review, we delineate all the common proprieties we find across the literature and how they relate to human interactions.
We identify two subcategories and highlight their characteristics and open challenges.
arXiv Detail & Related papers (2023-08-01T14:43:10Z) - Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism
of Language Models [49.39276272693035]
Large-scale pre-trained language models have shown remarkable memorizing ability.
Vanilla neural networks without pre-training have been long observed suffering from the catastrophic forgetting problem.
We find that 1) Vanilla language models are forgetful; 2) Pre-training leads to retentive language models; 3) Knowledge relevance and diversification significantly influence the memory formation.
arXiv Detail & Related papers (2023-05-16T03:50:38Z) - What Artificial Neural Networks Can Tell Us About Human Language
Acquisition [47.761188531404066]
Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language.
To increase the relevance of learnability results from computational models, we need to train model learners without significant advantages over humans.
arXiv Detail & Related papers (2022-08-17T00:12:37Z) - Same Neurons, Different Languages: Probing Morphosyntax in Multilingual
Pre-trained Models [84.86942006830772]
We conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar.
We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe.
arXiv Detail & Related papers (2022-05-04T12:22:31Z) - Creolizing the Web [2.393911349115195]
We present a method for detecting evolutionary patterns in a sociological model of language evolution.
We develop a minimalistic model that provides a rigorous base for any generalized evolutionary model for language based on communication between individuals.
We present empirical results and their interpretations on a real world dataset from rdt to identify communities and echo chambers for opinions.
arXiv Detail & Related papers (2021-02-24T16:08:45Z) - Emergent Multi-Agent Communication in the Deep Learning Era [26.764052787245728]
The ability to cooperate through language is a defining feature of humans.
As the perceptual, motory and planning capabilities of deep artificial networks increase, researchers are studying whether they also can develop a shared language to interact.
arXiv Detail & Related papers (2020-06-03T17:50:16Z) - Bridging Linguistic Typology and Multilingual Machine Translation with
Multi-View Language Representations [83.27475281544868]
We use singular vector canonical correlation analysis to study what kind of information is induced from each source.
We observe that our representations embed typology and strengthen correlations with language relationships.
We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
arXiv Detail & Related papers (2020-04-30T16:25:39Z) - Experience Grounds Language [185.73483760454454]
Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates.
Despite the incredible effectiveness of language processing models to tackle tasks after being trained on text alone, successful linguistic communication relies on a shared experience of the world.
arXiv Detail & Related papers (2020-04-21T16:56:27Z) - Co-evolution of language and agents in referential games [24.708802957946467]
We show that the optimal situation is to take into account the learning biases of the language learners and thus let language and agents co-evolve.
We pave the way to investigate the co-evolution of language in language emergence studies.
arXiv Detail & Related papers (2020-01-10T09:29:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.