From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition

• URL: http://arxiv.org/abs/2410.13259v1
• Date: Thu, 17 Oct 2024 06:31:49 GMT
• Title: From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition
• Authors: Qiyuan Yang, Pengda Wang, Luke D. Plonsky, Frederick L. Oswald, Hanjie Chen,
• Abstract summary: We propose a three-stage framework to assess the abilities of LMs. We evaluate the generative capacities of LMs using methods from linguistic research.
• Score: 6.617999710257379
• License:
• Abstract: We examine the language capabilities of language models (LMs) from the critical perspective of human language acquisition. Building on classical language development theories, we propose a three-stage framework to assess the abilities of LMs, ranging from preliminary word understanding to complex grammar and complex logical reasoning. Using this framework, we evaluate the generative capacities of LMs using methods from linguistic research. Results indicate that although recent LMs outperform earlier models in overall performance, their developmental trajectory does not strictly follow the path of human language acquisition. Notably, in generation tasks, LMs are more similar to human performance in areas where information is easier to extract from the corpus, such as average word length, clauses, and auxiliary verbs. Newer LMs did not exhibit significant progress in terms of specific dimensions, such as clauses and auxiliary verbs, where the variation across corpora is relatively limited. Register theory offers a plausible explanation for these observations, suggesting that the linguistic features of the training data have a substantial impact on the models' abilities.

Related papers

• Can Language Models Learn Typologically Implausible Languages? [62.823015163987996]
  Grammatical features across human languages show intriguing correlations often attributed to learning biases in humans. We discuss how language models (LMs) allow us to better determine the role of domain-general learning biases in language universals. We test LMs on an array of highly naturalistic but counterfactual versions of the English (head-initial) and Japanese (head-final) languages.
  arXiv  Detail & Related papers  (2025-02-17T20:40:01Z)
• Randomly Sampled Language Reasoning Problems Reveal Limits of LLMs [8.146860674148044]
  We attempt to measure models' language understanding capacity while circumventing the risk of dataset recall. We parameterize large families of language tasks recognized by deterministic finite automata (DFAs) We find that, even in the strikingly simple setting of 3-state DFAs, LLMs underperform un parameterized ngram models on both language recognition and synthesis tasks.
  arXiv  Detail & Related papers  (2025-01-06T07:57:51Z)
• The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model [59.357993924917]
  We study the evolution of multilingual capabilities in large language models (LLMs) during the pre-training process. We propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. We propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs.
  arXiv  Detail & Related papers  (2024-12-10T08:28:57Z)
• Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas [7.585433383340306]
  We show that tokenization-free, phoneme- and grapheme-based language models can achieve strong linguistic performance. Our findings suggest a promising direction for creating more linguistically plausible language models.
  arXiv  Detail & Related papers  (2024-10-02T12:36:08Z)
• Black Big Boxes: Do Language Models Hide a Theory of Adjective Order? [5.395055685742631]
  In English and other languages, multiple adjectives in a complex noun phrase show intricate ordering patterns that have been a target of much linguistic theory. We review existing hypotheses designed to explain Adjective Order Preferences (AOPs) in humans and develop a setup to study AOPs in language models. We find that all models' predictions are much closer to human AOPs than predictions generated by factors identified in theoretical linguistics.
  arXiv  Detail & Related papers  (2024-07-02T10:29:09Z)
• Holmes: A Benchmark to Assess the Linguistic Competence of Language Models [59.627729608055006]
  We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence. We use computation-based probing to examine LMs' internal representations regarding distinct linguistic phenomena. As a result, we meet recent calls to disentangle LMs' linguistic competence from other cognitive abilities.
  arXiv  Detail & Related papers  (2024-04-29T17:58:36Z)
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition [4.779196219827507]
  We argue that some of the most prominent benchmarks for evaluating the syntactic capacities of neural language models may not be sufficiently rigorous. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.
  arXiv  Detail & Related papers  (2023-10-31T00:16:17Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• A Survey of Large Language Models [81.06947636926638]
  Language modeling has been widely studied for language understanding and generation in the past two decades. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size.
  arXiv  Detail & Related papers  (2023-03-31T17:28:46Z)
• Shortcut Learning of Large Language Models in Natural Language Understanding [119.45683008451698]
  Large language models (LLMs) have achieved state-of-the-art performance on a series of natural language understanding tasks. They might rely on dataset bias and artifacts as shortcuts for prediction. This has significantly affected their generalizability and adversarial robustness.
  arXiv  Detail & Related papers  (2022-08-25T03:51:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.