Comparing LLM-generated and human-authored news text using formal syntactic theory

• URL: http://arxiv.org/abs/2506.01407v1
• Date: Mon, 02 Jun 2025 08:04:34 GMT
• Title: Comparing LLM-generated and human-authored news text using formal syntactic theory
• Abstract summary: This study provides the first comprehensive comparison of New York Times-style text generated by six large language models against real, human-authored NYT writing.<n>We use Head-driven Phrase Structure Grammar to analyze the grammatical structure of the texts.
• Score: 11.068340119264336
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study provides the first comprehensive comparison of New York Times-style text generated by six large language models against real, human-authored NYT writing. The comparison is based on a formal syntactic theory. We use Head-driven Phrase Structure Grammar (HPSG) to analyze the grammatical structure of the texts. We then investigate and illustrate the differences in the distributions of HPSG grammar types, revealing systematic distinctions between human and LLM-generated writing. These findings contribute to a deeper understanding of the syntactic behavior of LLMs as well as humans, within the NYT genre.

Related papers

• Does Syntax Matter? A Graph-Augmented Variational Topic Model for Computational Social Sciences [51.56484100374058]
  This paper introduces the Structural Contextual Probabilistic Topic Model ( SCPTM), an architecture that incorporates syntactic dependency relations into topic inference.<n>We evaluate seven topic modeling techniques across four corpora differing in register and discourse structure.<n>Results show that SCPTM's neural architecture yields substantial gains in document-topic alignment over generative baselines.
  arXiv  Detail & Related papers  (2026-09-07T17:39:06Z)
• More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs [6.96297373955807]
  This study contributes to a growing line of research in comparing LLM-generated texts with human-authored text.<n>We use diversity metrics from ecology and information theory to quantify variation in grammatical constructions and lexical types.<n>We show that English news text has changed little in the given time frame, while newer LLMs display reduced syntactic and, especially, lexical diversity compared to older, non-instruction-tuned models.
  arXiv  Detail & Related papers  (2026-05-07T11:21:29Z)
• Semantic Delta: An Interpretable Signal Differentiating Human and LLMs Dialogue [3.9431873099460826]
  We introduce a lightweight metric derived from semantic categories distribution.<n>We define semantic delta as the difference between the two most dominant category intensities within a dialogue.<n>Results show that AI-generated texts consistently produce higher deltas than human texts.
  arXiv  Detail & Related papers  (2026-03-20T11:01:47Z)
• Can Large Language Models (LLMs) Describe Pictures Like Children? A Comparative Corpus Study [0.0]
  This study evaluates how large language models (LLMs) replicate child-like language by comparing LLM-generated texts to a collection of German children's descriptions of picture stories.<n>We conducted a comparative analysis across psycholinguistic text properties, including word frequency, lexical richness, sentence and word length, part-of-speech tags, and semantic similarity with word embeddings.<n>The results show that LLM-generated texts are longer but less lexically rich, rely more on high-frequency words, and under-represent nouns.
  arXiv  Detail & Related papers  (2025-08-19T12:13:54Z)
• From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning [52.32745233116143]
  Humans organize knowledge into compact categories through semantic compression.<n>Large Language Models (LLMs) demonstrate remarkable linguistic abilities.<n>But whether their internal representations strike a human-like trade-off between compression and semantic fidelity is unclear.
  arXiv  Detail & Related papers  (2025-05-21T16:29:00Z)
• QUDsim: Quantifying Discourse Similarities in LLM-Generated Text [70.22275200293964]
  We introduce an abstraction based on linguistic theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progression.<n>We then use this framework to build $textbfQUDsim$, a similarity metric that can detect discursive parallels between documents.<n>Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) across samples, even when content differs.
  arXiv  Detail & Related papers  (2025-04-12T23:46:09Z)
• Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English [66.97110551643722]
  We investigate dialectal disparities in Large Language Models (LLMs) reasoning tasks.<n>We find that LLMs produce less accurate responses and simpler reasoning chains and explanations for AAE inputs.<n>These findings highlight systematic differences in how LLMs process and reason about different language varieties.
  arXiv  Detail & Related papers  (2025-03-06T05:15:34Z)
• Idiosyncrasies in Large Language Models [54.26923012617675]
  We unveil and study idiosyncrasies in Large Language Models (LLMs)<n>We find that fine-tuning existing text embedding models on LLM-generated texts yields excellent classification accuracy.<n>We leverage LLM as judges to generate detailed, open-ended descriptions of each model's idiosyncrasies.
  arXiv  Detail & Related papers  (2025-02-17T18:59:02Z)
• Large Language Models as Neurolinguistic Subjects: Discrepancy in Performance and Competence for Form and Meaning [49.60849499134362]
  This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning)<n>We introduce a neurolinguistic approach, utilizing a novel method that combines minimal pair and diagnostic probing to analyze activation patterns across model layers.<n>We found: (1) Psycholinguistic and neurolinguistic methods reveal that language performance and competence are distinct; (2) Direct probability measurement may not accurately assess linguistic competence; and (3) Instruction tuning won't change much competence but improve performance.
  arXiv  Detail & Related papers  (2024-11-12T04:16:44Z)
• Do LLMs write like humans? Variation in grammatical and rhetorical styles [0.7852714805965528]
  We study the rhetorical styles of large language models (LLMs) Using Douglas Biber's set of lexical, grammatical, and rhetorical features, we identify systematic differences between LLMs and humans. This demonstrates that despite their advanced abilities, LLMs struggle to match human styles.
  arXiv  Detail & Related papers  (2024-10-21T15:35:44Z)
• White Men Lead, Black Women Help? Benchmarking and Mitigating Language Agency Social Biases in LLMs [58.27353205269664]
  Social biases can manifest in language agency in Large Language Model (LLM)-generated content.<n>We introduce the Language Agency Bias Evaluation benchmark, which comprehensively evaluates biases in LLMs.<n>Using LABE, we unveil language agency social biases in 3 recent LLMs: ChatGPT, Llama3, and Mistral.
  arXiv  Detail & Related papers  (2024-04-16T12:27:54Z)
• Threads of Subtlety: Detecting Machine-Generated Texts Through Discourse Motifs [19.073560504913356]
  The line between human-crafted and machine-generated texts has become increasingly blurred. This paper delves into the inquiry of identifying discernible and unique linguistic properties in texts that were written by humans.
  arXiv  Detail & Related papers  (2024-02-16T11:20:30Z)
• Contrasting Linguistic Patterns in Human and LLM-Generated News Text [20.127243508644984]
  We conduct a quantitative analysis contrasting human-written English news text with comparable large language model (LLM) output. The results reveal various measurable differences between human and AI-generated texts. Human texts exhibit more scattered sentence length distributions, more variety of vocabulary, a distinct use of dependency and constituent types. LLM outputs use more numbers, symbols and auxiliaries than human texts, as well as more pronouns.
  arXiv  Detail & Related papers  (2023-08-17T15:54:38Z)
• Long-form analogies generated by chatGPT lack human-like psycholinguistic properties [0.5884031187931463]
  We apply psycholinguistic methods to evaluate individual sentences from long-form analogies about biochemical concepts. We compare analogies generated by human subjects enrolled in introductory biochemistry courses to analogies generated by chatGPT.
  arXiv  Detail & Related papers  (2023-06-07T15:42:31Z)
• The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative [7.03497683558609]
  Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. We present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC) Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning.
  arXiv  Detail & Related papers  (2022-10-24T13:01:24Z)
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN [63.79300884115027]
  Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? We introduce RAVEN, a suite of analyses for assessing the novelty of generated text.
  arXiv  Detail & Related papers  (2021-11-18T04:07:09Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.