From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition

• URL: http://arxiv.org/abs/2410.13259v1
• Date: Thu, 17 Oct 2024 06:31:49 GMT
• Title: From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition
• Authors: Qiyuan Yang, Pengda Wang, Luke D. Plonsky, Frederick L. Oswald, Hanjie Chen,
• Abstract summary: We propose a three-stage framework to assess the abilities of LMs. We evaluate the generative capacities of LMs using methods from linguistic research.
• Score: 6.617999710257379
• License:
• Abstract: We examine the language capabilities of language models (LMs) from the critical perspective of human language acquisition. Building on classical language development theories, we propose a three-stage framework to assess the abilities of LMs, ranging from preliminary word understanding to complex grammar and complex logical reasoning. Using this framework, we evaluate the generative capacities of LMs using methods from linguistic research. Results indicate that although recent LMs outperform earlier models in overall performance, their developmental trajectory does not strictly follow the path of human language acquisition. Notably, in generation tasks, LMs are more similar to human performance in areas where information is easier to extract from the corpus, such as average word length, clauses, and auxiliary verbs. Newer LMs did not exhibit significant progress in terms of specific dimensions, such as clauses and auxiliary verbs, where the variation across corpora is relatively limited. Register theory offers a plausible explanation for these observations, suggesting that the linguistic features of the training data have a substantial impact on the models' abilities.

Related papers

• Black Big Boxes: Do Language Models Hide a Theory of Adjective Order? [5.395055685742631]
  In English and other languages, multiple adjectives in a complex noun phrase show intricate ordering patterns that have been a target of much linguistic theory. We review existing hypotheses designed to explain Adjective Order Preferences (AOPs) in humans and develop a setup to study AOPs in language models. We find that all models' predictions are much closer to human AOPs than predictions generated by factors identified in theoretical linguistics.
  arXiv  Detail & Related papers  (2024-07-02T10:29:09Z)
• Holmes: A Benchmark to Assess the Linguistic Competence of Language Models [59.627729608055006]
  We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence. We use computation-based probing to examine LMs' internal representations regarding distinct linguistic phenomena. As a result, we meet recent calls to disentangle LMs' linguistic competence from other cognitive abilities.
  arXiv  Detail & Related papers  (2024-04-29T17:58:36Z)
• History, Development, and Principles of Large Language Models-An Introductory Survey [15.875687167037206]
  Language models serve as a cornerstone in natural language processing (NLP) Over extensive research spanning decades, language modeling has progressed from initial statistical language models (SLMs) to the contemporary landscape of large language models (LLMs)
  arXiv  Detail & Related papers  (2024-02-10T01:18:15Z)
• Evaluating Neural Language Models as Cognitive Models of Language Acquisition [4.779196219827507]
  We argue that some of the most prominent benchmarks for evaluating the syntactic capacities of neural language models may not be sufficiently rigorous. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.
  arXiv  Detail & Related papers  (2023-10-31T00:16:17Z)
• The Quo Vadis of the Relationship between Language and Large Language Models [3.10770247120758]
  Large Language Models (LLMs) have come to encourage the adoption of LLMs as scientific models of language. We identify the most important theoretical and empirical risks brought about by the adoption of scientific models that lack transparency. We conclude that, at their current stage of development, LLMs hardly offer any explanations for language.
  arXiv  Detail & Related papers  (2023-10-17T10:54:24Z)
• L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models [102.00201523306986]
  We present L2CEval, a systematic evaluation of the language-to-code generation capabilities of large language models (LLMs) We analyze the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs.
  arXiv  Detail & Related papers  (2023-09-29T17:57:00Z)
• A Survey of Large Language Models [81.06947636926638]
  Language modeling has been widely studied for language understanding and generation in the past two decades. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size.
  arXiv  Detail & Related papers  (2023-03-31T17:28:46Z)
• Augmented Language Models: a Survey [55.965967655575454]
  This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. We refer to them as Augmented Language Models (ALMs) The missing token objective allows ALMs to learn to reason, use tools, and even act, while still performing standard natural language tasks.
  arXiv  Detail & Related papers  (2023-02-15T18:25:52Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Shortcut Learning of Large Language Models in Natural Language Understanding [119.45683008451698]
  Large language models (LLMs) have achieved state-of-the-art performance on a series of natural language understanding tasks. They might rely on dataset bias and artifacts as shortcuts for prediction. This has significantly affected their generalizability and adversarial robustness.
  arXiv  Detail & Related papers  (2022-08-25T03:51:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.