What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure

• URL: http://arxiv.org/abs/2302.12239v4
• Date: Thu, 10 Oct 2024 11:43:58 GMT
• Title: What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
• Authors: Lukas Galke, Yoav Ram, Limor Raviv,
• Abstract summary: A fundamental property of language is its compositional structure, allowing humans to produce forms for new meanings. For humans, languages with more compositional and transparent structures are typically easier to learn than those with opaque and irregular structures. This learnability advantage has not yet been shown for deep neural networks, limiting their use as models for human language learning.
• Score: 5.871583927216651
• License:
• Abstract: Deep neural networks drive the success of natural language processing. A fundamental property of language is its compositional structure, allowing humans to systematically produce forms for new meanings. For humans, languages with more compositional and transparent structures are typically easier to learn than those with opaque and irregular structures. However, this learnability advantage has not yet been shown for deep neural networks, limiting their use as models for human language learning. Here, we directly test how neural networks compare to humans in learning and generalizing different languages that vary in their degree of compositional structure. We evaluate the memorization and generalization capabilities of a large language model and recurrent neural networks, and show that both deep neural networks exhibit a learnability advantage for more structured linguistic input: neural networks exposed to more compositional languages show more systematic generalization, greater agreement between different agents, and greater similarity to human learners.

Related papers

• Modeling rapid language learning by distilling Bayesian priors into artificial neural networks [18.752638142258668]
  We show that learning from limited naturalistic data is possible with an approach that combines the strong inductive biases of a Bayesian model with the flexible representations of a neural network. The resulting system can learn formal linguistic patterns from a small number of examples. It can also learn aspects of English syntax from a corpus of natural language.
  arXiv  Detail & Related papers  (2023-05-24T04:11:59Z)
• Nondeterministic Stacks in Neural Networks [0.456877715768796]
  We develop a differentiable data structure that efficiently simulates a nondeterministic pushdown automaton. We show that this raises their formal recognition power to arbitrary context-free languages. We also show that an RNN augmented with a nondeterministic stack is capable of surprisingly powerful behavior.
  arXiv  Detail & Related papers  (2023-04-25T16:00:40Z)
• Deep Learning Models to Study Sentence Comprehension in the Human Brain [0.1503974529275767]
  Recent artificial neural networks that process natural language achieve unprecedented performance in tasks requiring sentence-level understanding. We review works that compare these artificial language models with human brain activity and we assess the extent to which this approach has improved our understanding of the neural processes involved in natural language comprehension.
  arXiv  Detail & Related papers  (2023-01-16T10:31:25Z)
• What Artificial Neural Networks Can Tell Us About Human Language Acquisition [47.761188531404066]
  Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language. To increase the relevance of learnability results from computational models, we need to train model learners without significant advantages over humans.
  arXiv  Detail & Related papers  (2022-08-17T00:12:37Z)
• Dependency-based Mixture Language Models [53.152011258252315]
  We introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention.
  arXiv  Detail & Related papers  (2022-03-19T06:28:30Z)
• Emergence of Machine Language: Towards Symbolic Intelligence with Neural Networks [73.94290462239061]
  We propose to combine symbolism and connectionism principles by using neural networks to derive a discrete representation. By designing an interactive environment and task, we demonstrated that machines could generate a spontaneous, flexible, and semantic language.
  arXiv  Detail & Related papers  (2022-01-14T14:54:58Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)
• Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses [62.197912623223964]
  We show a low-dimensional structure where language models and translation models smoothly interpolate between word embeddings, syntactic and semantic tasks, and future word embeddings. We find that this representation embedding can predict how well each individual feature space maps to human brain responses to natural language stimuli recorded using fMRI. This suggests that the embedding captures some part of the brain's natural language representation structure.
  arXiv  Detail & Related papers  (2021-06-09T22:59:12Z)
• Compositional Processing Emerges in Neural Networks Solving Math Problems [100.80518350845668]
  Recent progress in artificial neural networks has shown that when large models are trained on enough linguistic data, grammatical structure emerges in their representations. We extend this work to the domain of mathematical reasoning, where it is possible to formulate precise hypotheses about how meanings should be composed. Our work shows that neural networks are not only able to infer something about the structured relationships implicit in their training data, but can also deploy this knowledge to guide the composition of individual meanings into composite wholes.
  arXiv  Detail & Related papers  (2021-05-19T07:24:42Z)
• Applications of Deep Neural Networks with Keras [0.0]
  Deep learning allows a neural network to learn hierarchies of information in a way that is like the function of the human brain. This course will introduce the student to classic neural network structures, Conversa Neural Networks (CNN), Long Short-Term Memory (LSTM), Gated Recurrent Neural Networks (GRU), General Adrial Networks (GAN)
  arXiv  Detail & Related papers  (2020-09-11T22:09:10Z)
• Compositional Languages Emerge in a Neural Iterated Learning Model [27.495624644227888]
  compositionality enables natural language to represent complex concepts via a structured combination of simpler ones. We propose an effective neural iterated learning (NIL) algorithm that, when applied to interacting neural agents, facilitates the emergence of a more structured type of language.
  arXiv  Detail & Related papers  (2020-02-04T15:19:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.