Is ChatGPT Transforming Academics' Writing Style?

• URL: http://arxiv.org/abs/2404.08627v2
• Date: Fri, 08 Nov 2024 18:56:42 GMT
• Title: Is ChatGPT Transforming Academics' Writing Style?
• Authors: Mingmeng Geng, Roberto Trotta,
• Abstract summary: Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT's writing style in their abstracts. We find that large language models (LLMs), represented by ChatGPT, are having an increasing impact on arXiv abstracts.
• Score: 0.0
• License:
• Abstract: Based on one million arXiv papers submitted from May 2018 to January 2024, we assess the textual density of ChatGPT's writing style in their abstracts through a statistical analysis of word frequency changes. Our model is calibrated and validated on a mixture of real abstracts and ChatGPT-modified abstracts (simulated data) after a careful noise analysis. The words used for estimation are not fixed but adaptive, including those with decreasing frequency. We find that large language models (LLMs), represented by ChatGPT, are having an increasing impact on arXiv abstracts, especially in the field of computer science, where the fraction of LLM-style abstracts is estimated to be approximately 35%, if we take the responses of GPT-3.5 to one simple prompt, "revise the following sentences", as a baseline. We conclude with an analysis of both positive and negative aspects of the penetration of LLMs into academics' writing style.

Related papers

• Evaluating the Predictive Capacity of ChatGPT for Academic Peer Review Outcomes Across Multiple Platforms [3.3543455244780223]
  This paper introduces two new contexts and employs a more robust method - averaging multiple ChatGPT scores. Findings that averaging 30 ChatGPT predictions, based on reviewer guidelines and using only submitted titles and abstracts, failed to predict peer review outcomes for F1000Research.
  arXiv  Detail & Related papers  (2024-11-14T19:20:33Z)
• Impact of ChatGPT on the writing style of condensed matter physicists [6.653378613306849]
  We estimate the impact of ChatGPT's release on the writing style of condensed matter papers on arXiv. Our analysis reveals a statistically significant improvement in the English quality of abstracts written by non-native English speakers.
  arXiv  Detail & Related papers  (2024-08-30T14:37:10Z)
• Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs [3.9627148816681284]
  This article assesses which ChatGPT inputs produce better quality score estimates. The optimal input is the article title and abstract, with average ChatGPT scores based on these correlating at 0.67 with human scores.
  arXiv  Detail & Related papers  (2024-08-13T09:19:21Z)
• Delving into ChatGPT usage in academic writing through excess vocabulary [4.58733012283457]
  Large language models (LLMs) can generate and revise text with human-level performance. Yet, many scientists have been using them to assist their scholarly writing. We study vocabulary changes in 14 million PubMed abstracts from 2010-2024, and show how the appearance of LLMs led to an abrupt increase in the frequency of certain style words.
  arXiv  Detail & Related papers  (2024-06-11T07:16:34Z)
• Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews [51.453135368388686]
  We present an approach for estimating the fraction of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM) Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently examine real-world LLM-use at the corpus level.
  arXiv  Detail & Related papers  (2024-03-11T21:51:39Z)
• Understanding writing style in social media with a supervised contrastively pre-trained transformer [57.48690310135374]
  Online Social Networks serve as fertile ground for harmful behavior, ranging from hate speech to the dissemination of disinformation. We introduce the Style Transformer for Authorship Representations (STAR), trained on a large corpus derived from public sources of 4.5 x 106 authored texts. Using a support base of 8 documents of 512 tokens, we can discern authors from sets of up to 1616 authors with at least 80% accuracy.
  arXiv  Detail & Related papers  (2023-10-17T09:01:17Z)
• Evaluating Language Models for Mathematics through Interactions [116.67206980096513]
  We introduce CheckMate, a prototype platform for humans to interact with and evaluate large language models (LLMs) We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics. We derive a taxonomy of human behaviours and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness.
  arXiv  Detail & Related papers  (2023-06-02T17:12:25Z)
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models [55.60306377044225]
  "SelfCheckGPT" is a simple sampling-based approach to fact-check the responses of black-box models. We investigate this approach by using GPT-3 to generate passages about individuals from the WikiBio dataset.
  arXiv  Detail & Related papers  (2023-03-15T19:31:21Z)
• Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization [28.104696513516117]
  Large language models (LLMs) like GPT3 and ChatGPT have recently created significant interest in using these models for text summarization tasks. Recent studies citegoyal2022news, zhang2023benchmarking have shown that LLMs-generated news summaries are already on par with humans. Our experiments reveal that ChatGPT's performance is comparable to traditional fine-tuning methods in terms of Rouge scores.
  arXiv  Detail & Related papers  (2023-02-16T04:41:30Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)
• Investigating Pretrained Language Models for Graph-to-Text Generation [55.55151069694146]
  Graph-to-text generation aims to generate fluent texts from graph-based data. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further.
  arXiv  Detail & Related papers  (2020-07-16T16:05:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.