Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment

• URL: http://arxiv.org/abs/2005.00165v3
• Date: Thu, 7 May 2020 15:21:58 GMT
• Title: Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment
• Authors: Forrest Davis and Marten van Schijndel
• Abstract summary: We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences.
• Score: 17.995905582226463
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A standard approach to evaluating language models analyzes how models assign probabilities to valid versus invalid syntactic constructions (i.e. is a grammatical sentence more probable than an ungrammatical sentence). Our work uses ambiguous relative clause attachment to extend such evaluations to cases of multiple simultaneous valid interpretations, where stark grammaticality differences are absent. We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. Thus, English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences. We conclude by relating these results to broader concerns about the relationship between comprehension (i.e. typical language model use cases) and production (which generates the training data for language models), suggesting that necessary linguistic biases are not present in the training signal at all.

Related papers

• JCoLA: Japanese Corpus of Linguistic Acceptability [3.6141428739228902]
  We introduce JCoLA (Japanese Corpus of Linguistic Acceptability), which consists of 10,020 sentences annotated with binary acceptability judgments. We then evaluate the syntactic knowledge of 9 different types of Japanese language models on JCoLA.
  arXiv  Detail & Related papers  (2023-09-22T07:35:45Z)
• Language model acceptability judgements are not always robust to context [30.868765627701457]
  We investigate the stability of language models' performance on targeted syntactic evaluations. We find that model judgements are generally robust when placed in randomly sampled linguistic contexts. We show that these changes in model performance are not explainable by simple features matching the context and the test inputs.
  arXiv  Detail & Related papers  (2022-12-18T00:11:06Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• On The Ingredients of an Effective Zero-shot Semantic Parser [95.01623036661468]
  We analyze zero-shot learning by paraphrasing training examples of canonical utterances and programs from a grammar. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods. Our model achieves strong performance on two semantic parsing benchmarks (Scholar, Geo) with zero labeled data.
  arXiv  Detail & Related papers  (2021-10-15T21:41:16Z)
• Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models [22.57309958548928]
  We investigate whether structural supervision improves language models' ability to learn grammatical dependencies in typologically different languages. We train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and generative parsing models on datasets of different sizes. We find suggestive evidence that structural supervision helps with representing syntactic state across intervening content and improves performance in low-data settings.
  arXiv  Detail & Related papers  (2021-09-22T22:11:30Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Understanding by Understanding Not: Modeling Negation in Language Models [81.21351681735973]
  Negation is a core construction in natural language. We propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences. We reduce the mean top1 error rate to 4% on the negated LAMA dataset.
  arXiv  Detail & Related papers  (2021-05-07T21:58:35Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Cross-Linguistic Syntactic Evaluation of Word Prediction Models [25.39896327641704]
  We investigate how neural word prediction models' ability to learn syntax varies by language. CLAMS includes subject-verb agreement challenge sets for English, French, German, Hebrew and Russian. We use CLAMS to evaluate LSTM language models as well as monolingual and multilingual BERT.
  arXiv  Detail & Related papers  (2020-05-01T02:51:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.