More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs

• URL: http://arxiv.org/abs/2605.06030v1
• Date: Thu, 07 May 2026 11:21:29 GMT
• Title: More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs
• Abstract summary: This study contributes to a growing line of research in comparing LLM-generated texts with human-authored text.<n>We use diversity metrics from ecology and information theory to quantify variation in grammatical constructions and lexical types.<n>We show that English news text has changed little in the given time frame, while newer LLMs display reduced syntactic and, especially, lexical diversity compared to older, non-instruction-tuned models.
• Score: 6.96297373955807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study contributes to a growing line of research in comparing LLM-generated texts with human-authored text, in this case, English news text. We focus in particular on the evaluation of syntactic properties through formal grammar frameworks. Our analysis compares two generations of LLMs in the context of two human-authored English news datasets from two different years. Employing the Head-Driven Phrase Structure Grammar (HPSG) formalism, we investigate the distributions of syntactic structures and lexical types of AI-generated texts and contrast them with the corresponding distributions in the human-authored New York Times (NYT) articles. We use diversity metrics from ecology and information theory to quantify variation in grammatical constructions and lexical types. We show that English news text has changed little in the given time frame, while newer LLMs display reduced syntactic and, especially, lexical diversity compared to older, non-instruction-tuned models. These findings point to future work in studying effects of instruction tuning, which, while enhancing coherence and adherence to prompts, may narrow the expressive range of model output.

Related papers

• On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generation [79.18001266925245]
  LingT2I is a benchmark covering 10 widely used languages with 33K prompts.<n>We conduct a comprehensive cross-lingual analysis, uncovering linguistic inequality and language-dependent trade-offs.
  arXiv  Detail & Related papers  (2026-08-11T14:49:23Z)
• Comparing LLM-generated and human-authored news text using formal syntactic theory [11.068340119264336]
  This study provides the first comprehensive comparison of New York Times-style text generated by six large language models against real, human-authored NYT writing.<n>We use Head-driven Phrase Structure Grammar to analyze the grammatical structure of the texts.
  arXiv  Detail & Related papers  (2025-06-02T08:04:34Z)
• Divergent LLM Adoption and Heterogeneous Convergence Paths in Research Writing [0.8046044493355781]
  Large Language Models (LLMs) are reshaping content creation and academic writing.<n>This study investigates the impact of AI-assisted generative revisions on research manuscripts.
  arXiv  Detail & Related papers  (2025-04-18T11:09:16Z)
• The dynamics of meaning through time: Assessment of Large Language Models [2.5864824580604515]
  This study aims to evaluate the capabilities of various large language models (LLMs) in capturing temporal dynamics of meaning.<n>Our comparative analysis includes prominent models like ChatGPT, GPT-4, Claude, Bard, Gemini, and Llama.<n>Findings reveal marked differences in each model's handling of historical context and semantic shifts, highlighting both strengths and limitations in temporal semantic understanding.
  arXiv  Detail & Related papers  (2025-01-09T19:56:44Z)
• Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models [0.0]
  We identify significant differences between human-written texts and those generated by large language models (LLMs)<n>Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs.
  arXiv  Detail & Related papers  (2024-12-04T04:38:35Z)
• Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competence [49.60849499134362]
  This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning)<n>We introduce a neurolinguistic approach, utilizing a novel method that combines minimal pair and diagnostic probing to analyze activation patterns across model layers.<n>We found: (1) Psycholinguistic and neurolinguistic methods reveal that language performance and competence are distinct; (2) Direct probability measurement may not accurately assess linguistic competence; and (3) Instruction tuning won't change much competence but improve performance.
  arXiv  Detail & Related papers  (2024-11-12T04:16:44Z)
• Inclusivity in Large Language Models: Personality Traits and Gender Bias in Scientific Abstracts [49.97673761305336]
  We evaluate three large language models (LLMs) for their alignment with human narrative styles and potential gender biases. Our findings indicate that, while these models generally produce text closely resembling human authored content, variations in stylistic features suggest significant gender biases.
  arXiv  Detail & Related papers  (2024-06-27T19:26:11Z)
• Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis [2.2530496464901106]
  We evaluate semantic representations of Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models.<n>We find that various BERT-based LMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark.
  arXiv  Detail & Related papers  (2024-06-20T18:58:11Z)
• Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning [57.74233319453229]
  Large language models (LLMs) have emerged as a groundbreaking technology and their unparalleled text generation capabilities have sparked interest in their application to the fundamental sentence representation learning task. We propose MultiCSR, a multi-level contrastive sentence representation learning framework that decomposes the process of prompting LLMs to generate a corpus. Our experiments reveal that MultiCSR enables a less advanced LLM to surpass the performance of ChatGPT, while applying it to ChatGPT achieves better state-of-the-art results.
  arXiv  Detail & Related papers  (2023-10-17T03:21:43Z)
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN [63.79300884115027]
  Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? We introduce RAVEN, a suite of analyses for assessing the novelty of generated text.
  arXiv  Detail & Related papers  (2021-11-18T04:07:09Z)
• Dependency Induction Through the Lens of Visual Perception [81.91502968815746]
  We propose an unsupervised grammar induction model that leverages word concreteness and a structural vision-based to jointly learn constituency-structure and dependency-structure grammars. Our experiments show that the proposed extension outperforms the current state-of-the-art visually grounded models in constituency parsing even with a smaller grammar size.
  arXiv  Detail & Related papers  (2021-09-20T18:40:37Z)
• A Comparative Study of Lexical Substitution Approaches based on Neural Language Models [117.96628873753123]
  We present a large-scale comparative study of popular neural language and masked language models. We show that already competitive results achieved by SOTA LMs/MLMs can be further improved if information about the target word is injected properly.
  arXiv  Detail & Related papers  (2020-05-29T18:43:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.