Word Frequency Does Not Predict Grammatical Knowledge in Language Models

• URL: http://arxiv.org/abs/2010.13870v1
• Date: Mon, 26 Oct 2020 19:51:36 GMT
• Title: Word Frequency Does Not Predict Grammatical Knowledge in Language Models
• Authors: Charles Yu, Ryan Sie, Nico Tedeschi, Leon Bergen
• Abstract summary: We investigate whether there are systematic sources of variation in the language models' accuracy. We find that certain nouns are systematically understood better than others, an effect which is robust across grammatical tasks and different language models. We find that a novel noun's grammatical properties can be few-shot learned from various types of training data.
• Score: 2.1984302611206537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural language models learn, to varying degrees of accuracy, the grammatical properties of natural languages. In this work, we investigate whether there are systematic sources of variation in the language models' accuracy. Focusing on subject-verb agreement and reflexive anaphora, we find that certain nouns are systematically understood better than others, an effect which is robust across grammatical tasks and different language models. Surprisingly, we find that across four orders of magnitude, corpus frequency is unrelated to a noun's performance on grammatical tasks. Finally, we find that a novel noun's grammatical properties can be few-shot learned from various types of training data. The results present a paradox: there should be less variation in grammatical performance than is actually observed.

Related papers

• Principles of semantic and functional efficiency in grammatical patterning [1.6267479602370545]
  Grammatical features such as number and gender serve two central functions in human languages. Number and gender encode salient semantic attributes like numerosity and animacy, but offload sentence processing cost by predictably linking words together. Grammars exhibit consistent organizational patterns across diverse languages, invariably rooted in a semantic foundation.
  arXiv  Detail & Related papers  (2024-10-21T10:49:54Z)
• What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languages [51.0349882045866]
  This paper investigates biases of Large Language Models (LLMs) through the lens of grammatical gender. We prompt a model to describe nouns with adjectives in various languages, focusing specifically on languages with grammatical gender. We find that a simple classifier can not only predict noun gender above chance but also exhibit cross-language transferability.
  arXiv  Detail & Related papers  (2024-07-12T22:10:16Z)
• Syntactic Variation Across the Grammar: Modelling a Complex Adaptive System [0.76146285961466]
  We model dialectal variation across 49 local populations of English speakers in 16 countries. Results show that an important part of syntactic variation consists of interactions between different parts of the grammar. New Zealand English could be more similar to Australian English in phrasal verbs but at the same time more similar to UK English in dative phrases.
  arXiv  Detail & Related papers  (2023-09-21T08:14:34Z)
• Quantifying the Roles of Visual, Linguistic, and Visual-Linguistic Complexity in Verb Acquisition [8.183763443800348]
  We employ visual and linguistic representations of words sourced from pre-trained artificial neural networks. We find that the representation of verbs is generally more variable and less discriminable within domain than the representation of nouns. Visual variability is the strongest factor that internally drives verb learning, followed by visual-linguistic alignment and linguistic variability.
  arXiv  Detail & Related papers  (2023-04-05T15:08:21Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change [6.7485485663645495]
  We first compare the performance of grammatical profiles against that of a multilingual neural language model (XLM-R) on 10 datasets, covering 7 languages. Our results show that ensembling grammatical profiles with XLM-R improves semantic change detection performance for most datasets and languages.
  arXiv  Detail & Related papers  (2022-04-12T11:20:42Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model [66.84264870118723]
  We present the first purely corpus-driven model of multi-lingual adjective ordering in the form of a latent-variable model. We provide strong converging evidence for the existence of universal, cross-linguistic, hierarchical adjective ordering tendencies.
  arXiv  Detail & Related papers  (2020-10-09T18:27:55Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)
• An exploration of the encoding of grammatical gender in word embeddings [0.6461556265872973]
  The study of grammatical gender based on word embeddings can give insight into discussions on how grammatical genders are determined. It is found that there is an overlap in how grammatical gender is encoded in Swedish, Danish, and Dutch embeddings.
  arXiv  Detail & Related papers  (2020-08-05T06:01:46Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.