Are Large Language Models a Threat to Digital Public Goods? Evidence
from Activity on Stack Overflow
- URL: http://arxiv.org/abs/2307.07367v1
- Date: Fri, 14 Jul 2023 14:22:12 GMT
- Title: Are Large Language Models a Threat to Digital Public Goods? Evidence
from Activity on Stack Overflow
- Authors: Maria del Rio-Chanona, Nadzeya Laurentsyeva, Johannes Wachs
- Abstract summary: We investigate how the release of ChatGPT changed human-generated open data on the web.
We find that relative to its Russian and Chinese counterparts, where access to ChatGPT is limited, activity on Stack Overflow significantly decreased.
- Score: 1.5039745292757671
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large language models like ChatGPT efficiently provide users with information
about various topics, presenting a potential substitute for searching the web
and asking people for help online. But since users interact privately with the
model, these models may drastically reduce the amount of publicly available
human-generated data and knowledge resources. This substitution can present a
significant problem in securing training data for future models. In this work,
we investigate how the release of ChatGPT changed human-generated open data on
the web by analyzing the activity on Stack Overflow, the leading online Q\&A
platform for computer programming. We find that relative to its Russian and
Chinese counterparts, where access to ChatGPT is limited, and to similar forums
for mathematics, where ChatGPT is less capable, activity on Stack Overflow
significantly decreased. A difference-in-differences model estimates a 16\%
decrease in weekly posts on Stack Overflow. This effect increases in magnitude
over time, and is larger for posts related to the most widely used programming
languages. Posts made after ChatGPT get similar voting scores than before,
suggesting that ChatGPT is not merely displacing duplicate or low-quality
content. These results suggest that more users are adopting large language
models to answer questions and they are better substitutes for Stack Overflow
for languages for which they have more training data. Using models like ChatGPT
may be more efficient for solving certain programming problems, but its
widespread adoption and the resulting shift away from public exchange on the
web will limit the open data people and models can learn from in the future.
Related papers
- Remote Timing Attacks on Efficient Language Model Inference [63.79839291641793]
We show it is possible to exploit timing differences to mount a timing attack.
We show how it is possible to learn the topic of a user's conversation with 90%+ precision.
An adversary can leverage a boosting attack to recover PII placed in messages for open source systems.
arXiv Detail & Related papers (2024-10-22T16:51:36Z) - Unmasking the giant: A comprehensive evaluation of ChatGPT's proficiency in coding algorithms and data structures [0.6990493129893112]
We evaluate ChatGPT's ability to generate correct solutions to the problems fed to it, its code quality, and nature of run-time errors thrown by its code.
We look into patterns in the test cases passed in order to gain some insights into how wrong ChatGPT code is in these kinds of situations.
arXiv Detail & Related papers (2023-07-10T08:20:34Z) - The Curse of Recursion: Training on Generated Data Makes Models Forget [70.02793975243212]
Large language models (LLMs) are here to stay, and will bring about drastic change in the whole ecosystem of online text and images.
We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear.
arXiv Detail & Related papers (2023-05-27T15:10:41Z) - ChatGPT to Replace Crowdsourcing of Paraphrases for Intent
Classification: Higher Diversity and Comparable Model Robustness [3.126776200660494]
We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models.
Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks.
arXiv Detail & Related papers (2023-05-22T11:46:32Z) - Phoenix: Democratizing ChatGPT across Languages [68.75163236421352]
We release a large language model "Phoenix", achieving competitive performance among open-source English and Chinese models.
We believe this work will be beneficial to make ChatGPT more accessible, especially in countries where people cannot use ChatGPT due to restrictions from OpenAI or local goverments.
arXiv Detail & Related papers (2023-04-20T16:50:04Z) - ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large
Language Models in Multilingual Learning [70.57126720079971]
Large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP)
This paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources.
Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages.
arXiv Detail & Related papers (2023-04-12T05:08:52Z) - ChatGPT-Crawler: Find out if ChatGPT really knows what it's talking
about [15.19126287569545]
This research examines the responses generated by ChatGPT from different Conversational QA corpora.
The study employed BERT similarity scores to compare these responses with correct answers and obtain Natural Language Inference(NLI) labels.
The study identified instances where ChatGPT provided incorrect answers to questions, providing insights into areas where the model may be prone to error.
arXiv Detail & Related papers (2023-04-06T18:42:47Z) - Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot.
Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community.
It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
arXiv Detail & Related papers (2023-02-08T09:44:51Z) - A Categorical Archive of ChatGPT Failures [47.64219291655723]
ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation.
It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries.
However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study.
arXiv Detail & Related papers (2023-02-06T04:21:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.