How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses

• URL: http://arxiv.org/abs/2504.13038v1
• Date: Thu, 17 Apr 2025 15:51:59 GMT
• Title: How Large Language Models Are Changing MOOC Essay Answers: A Comparison of Pre- and Post-LLM Responses
• Authors: Leo Leppänen, Lili Aunimo, Arto Hellas, Jukka K. Nurminen, Linda Mannila,
• Abstract summary: We analyze essay responses from a free university-level MOOC on AI ethics.<n>We find that the launch of ChatGPT coincided with significant changes in both the length and style of student essays.<n>We also observe -- as expected based on related public discourse -- changes in prevalence of key content words related to AI and LLMs.
• Score: 2.681175517296992
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The release of ChatGPT in late 2022 caused a flurry of activity and concern in the academic and educational communities. Some see the tool's ability to generate human-like text that passes at least cursory inspections for factual accuracy ``often enough'' a golden age of information retrieval and computer-assisted learning. Some, on the other hand, worry the tool may lead to unprecedented levels of academic dishonesty and cheating. In this work, we quantify some of the effects of the emergence of Large Language Models (LLMs) on online education by analyzing a multi-year dataset of student essay responses from a free university-level MOOC on AI ethics. Our dataset includes essays submitted both before and after ChatGPT's release. We find that the launch of ChatGPT coincided with significant changes in both the length and style of student essays, mirroring observations in other contexts such as academic publishing. We also observe -- as expected based on related public discourse -- changes in prevalence of key content words related to AI and LLMs, but not necessarily the general themes or topics discussed in the student essays as identified through (dynamic) topic modeling.

Related papers

• What Evidence Do Language Models Find Convincing? [94.90663008214918]
  We build a dataset that pairs controversial queries with a series of real-world evidence documents that contain different facts. We use this dataset to perform sensitivity and counterfactual analyses to explore which text features most affect LLM predictions. Overall, we find that current models rely heavily on the relevance of a website to the query, while largely ignoring stylistic features that humans find important.
  arXiv  Detail & Related papers  (2024-02-19T02:15:34Z)
• Chatbot-supported Thesis Writing: An Autoethnographic Report [0.0]
  ChatGPT might be applied to formats that require learners to generate text, such as bachelor theses or student research papers. ChatGPT is to be valued as a beneficial tool in thesis writing. However, writing a conclusive thesis still requires the learner's meaningful engagement.
  arXiv  Detail & Related papers  (2023-10-14T09:09:26Z)
• ChatGPT impacts in programming education: A recent literature overview that debates ChatGPT responses [0.0]
  It lists, covers and documents from the literature the major issues that have been identified for this topic. The paper ends with a short discussion on what is expected to happen in the near future.
  arXiv  Detail & Related papers  (2023-08-30T06:41:57Z)
• AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models [74.10293412011455]
  We propose AutoConv for synthetic conversation generation. Specifically, we formulate the conversation generation problem as a language modeling task. We finetune an LLM with a few human conversations to capture the characteristics of the information-seeking process.
  arXiv  Detail & Related papers  (2023-08-12T08:52:40Z)
• SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents [72.42049370297849]
  SpokenWOZ is a large-scale speech-text dataset for spoken TOD. Cross-turn slot and reasoning slot detection are new challenges for SpokenWOZ.
  arXiv  Detail & Related papers  (2023-05-22T13:47:51Z)
• ChatGPT in the Classroom: An Analysis of Its Strengths and Weaknesses for Solving Undergraduate Computer Science Questions [5.962828109329824]
  ChatGPT is an AI language model developed by OpenAI that can understand and generate human-like text. There is concern that students may leverage ChatGPT to complete take-home assignments and exams and obtain favorable grades without genuinely acquiring knowledge.
  arXiv  Detail & Related papers  (2023-04-28T17:26:32Z)
• AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays [66.36541161082856]
  ChatGPT and similar generative AI models have attracted hundreds of millions of users. This study compares human-written versus ChatGPT-generated argumentative student essays.
  arXiv  Detail & Related papers  (2023-04-24T12:58:28Z)
• Will ChatGPT get you caught? Rethinking of Plagiarism Detection [0.0]
  The rise of Artificial Intelligence (AI) technology and its impact on education has been a topic of growing concern in recent years. The use of chatbots, particularly ChatGPT, for generating academic essays has sparked fears among scholars. This study aims to explore the originality of contents produced by one of the most popular AI chatbots, ChatGPT.
  arXiv  Detail & Related papers  (2023-02-08T20:59:18Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.