Related papers: Injecting structural hints: Using language models to study inductive biases in language learning

Injecting structural hints: Using language models to study inductive biases in language learning

URL: http://arxiv.org/abs/2304.13060v2
Date: Sun, 29 Oct 2023 17:14:06 GMT
Title: Injecting structural hints: Using language models to study inductive biases in language learning
Authors: Isabel Papadimitriou and Dan Jurafsky
Abstract summary: We inject inductive bias into language models by pretraining on formally-structured data. We then evaluate the biased learners' ability to learn typologically-diverse natural languages. We show that non-context-free relationships form the best inductive biases.
Score: 40.8902073270634
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Both humans and large language models are able to learn language without explicit structural supervision. What inductive biases make this learning possible? We address this fundamental cognitive question by leveraging transformer language models: we inject inductive bias into language models by pretraining on formally-structured data, and then evaluate the biased learners' ability to learn typologically-diverse natural languages. Our experimental setup creates a testbed for hypotheses about inductive bias in human language learning. We investigate the effect of injecting models with three types of inductive bias: 1) recursive, hierarchical processing, 2) crossing token-token relationships that can't be modeled by context-free grammars, and 3) a Zipfian power-law vocabulary distribution. We show that non-context-free relationships form the best inductive biases. Our study leverages the capabilities of transformer models to run controlled language learning experiments that are not possible to run on humans, and surfaces hypotheses about the structures that facilitate language learning in both humans and machines.