What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

• URL: http://arxiv.org/abs/2605.19936v1
• Date: Tue, 19 May 2026 14:54:33 GMT
• Title: What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience
• Abstract summary: We create a naturalistic corpus of over 37,000 papers from the ACL Anthology ( 2020-2024) and a synthetic dataset of 3,000 human-written passages.<n>We first implement a series of diachronic lexical analyses, showing that both word frequency and usage contexts have changed significantly over time.<n>We then model a range of more complex stylistic features and find that LLM-modified texts more frequently contain certain syntactic constructions, more complex and longer words and a lower lexical diversity.
• Score: 8.092815324198176
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Has the style of scientific communication changed due to the growing use of large language models in the writing process? We address this question in the domain of Natural Language Processing by leveraging two data resources we create: a naturalistic corpus of over 37,000 papers from the ACL Anthology (2020-2024); and a synthetic dataset of 3,000 human-written passages and their LLM-generated improvements. We first implement a series of diachronic lexical analyses, showing that both word frequency and usage contexts have changed significantly over time, indicating semantic specialization in some cases and generalization in others. Broadening our perspective, we then model a range of more complex stylistic features and find that LLM-modified texts more frequently contain certain syntactic constructions, more complex and longer words and a lower lexical diversity. Finally, we connect these changes in writing practices to subjective reading experience through a pilot annotation study with 20 domain experts. They overall rate LLM-improved texts as more understandable and exciting, but also express negative qualitative attitudes towards LLMs, highlighting the strongly subjective effect of AI-assisted writing on reading experience.

Related papers

• More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs [6.96297373955807]
  This study contributes to a growing line of research in comparing LLM-generated texts with human-authored text.<n>We use diversity metrics from ecology and information theory to quantify variation in grammatical constructions and lexical types.<n>We show that English news text has changed little in the given time frame, while newer LLMs display reduced syntactic and, especially, lexical diversity compared to older, non-instruction-tuned models.
  arXiv  Detail & Related papers  (2026-05-07T11:21:29Z)
• CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Models [2.872898284494118]
  We focus on argumentative texts and their improvement, a task named Argument Improvement (ArgImp)<n>We present CLEAR: an evaluation pipeline consisting of 57 metrics mapped to four linguistic levels.<n>We find that the models perform ArgImp by shortening the texts while simultaneously increasing average word length and merging sentences.
  arXiv  Detail & Related papers  (2025-09-18T14:53:41Z)
• How Do LLM-Generated Texts Impact Term-Based Retrieval Models? [76.92519309816008]
  This paper investigates the influence of large language models (LLMs) on term-based retrieval models.<n>Our linguistic analysis reveals that LLM-generated texts exhibit smoother high-frequency and steeper low-frequency Zipf slopes.<n>Our study further explores whether term-based retrieval models demonstrate source bias, concluding that these models prioritize documents whose term distributions closely correspond to those of the queries.
  arXiv  Detail & Related papers  (2025-08-25T06:43:27Z)
• Examining Linguistic Shifts in Academic Writing Before and After the Launch of ChatGPT: A Study on Preprint Papers [7.161475868971293]
  Large Language Models (LLMs) have prompted academic concerns about their impact on academic writing.<n>We conducted a large-scale analysis across 823,798 abstracts published in last decade from arXiv dataset.
  arXiv  Detail & Related papers  (2025-05-18T03:35:43Z)
• Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs? [2.3749120526936465]
  This study explores how recent large language models (LLMs) navigate relative clause attachment ambiguity in six typologically diverse languages.
  arXiv  Detail & Related papers  (2025-03-13T19:44:15Z)
• Beyond Profile: From Surface-Level Facts to Deep Persona Simulation in LLMs [50.0874045899661]
  We introduce CharacterBot, a model designed to replicate both the linguistic patterns and distinctive thought patterns as manifested in the textual works of a character.<n>Using Lu Xun, a renowned Chinese writer as a case study, we propose four training tasks derived from his 17 essay collections.<n>These include a pre-training task focused on mastering external linguistic structures and knowledge, as well as three fine-tuning tasks.<n>We evaluate CharacterBot on three tasks for linguistic accuracy and opinion comprehension, demonstrating that it significantly outperforms the baselines on our adapted metrics.
  arXiv  Detail & Related papers  (2025-02-18T16:11:54Z)
• Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models [0.0]
  We identify significant differences between human-written texts and those generated by large language models (LLMs)<n>Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs.
  arXiv  Detail & Related papers  (2024-12-04T04:38:35Z)
• Inclusivity in Large Language Models: Personality Traits and Gender Bias in Scientific Abstracts [49.97673761305336]
  We evaluate three large language models (LLMs) for their alignment with human narrative styles and potential gender biases. Our findings indicate that, while these models generally produce text closely resembling human authored content, variations in stylistic features suggest significant gender biases.
  arXiv  Detail & Related papers  (2024-06-27T19:26:11Z)
• Mapping the Increasing Use of LLMs in Scientific Papers [99.67983375899719]
  We conduct the first systematic, large-scale analysis across 950,965 papers published between January 2020 and February 2024 on the arXiv, bioRxiv, and Nature portfolio journals. Our findings reveal a steady increase in LLM usage, with the largest and fastest growth observed in Computer Science papers.
  arXiv  Detail & Related papers  (2024-04-01T17:45:15Z)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews [51.453135368388686]
  We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM) Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level.
  arXiv  Detail & Related papers  (2024-03-11T21:51:39Z)
• Improving Factual Consistency of News Summarization by Contrastive Preference Optimization [65.11227166319546]
  Large language models (LLMs) generate summaries that are factually inconsistent with original articles.<n>These hallucinations are challenging to detect through traditional methods.<n>We propose Contrastive Preference Optimization (CPO) to disentangle the LLMs' propensities to generate faithful and fake content.
  arXiv  Detail & Related papers  (2023-10-30T08:40:16Z)
• Cross-Lingual Knowledge Editing in Large Language Models [73.12622532088564]
  Knowledge editing has been shown to adapt large language models to new knowledge without retraining from scratch. It is still unknown the effect of source language editing on a different target language. We first collect a large-scale cross-lingual synthetic dataset by translating ZsRE from English to Chinese.
  arXiv  Detail & Related papers  (2023-09-16T11:07:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.