Can Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection

• URL: http://arxiv.org/abs/2606.04924v1
• Date: Wed, 03 Jun 2026 14:18:27 GMT
• Title: Can Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection
• Abstract summary: The widespread use of Large Language Models (LLMs) as writing tools challenges the validity of crowdsourced data.<n>This paper provides an overview of practitioners' challenges, mitigation strategies, and the foreseen implications on data quality.
• Score: 8.623161636452592
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The widespread use of Large Language Models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models. To better understand how this is addressed, we surveyed 155 researchers in NLP and related disciplines about their experiences and opinions on collecting free-text responses via crowdsourcing. This paper provides an overview of practitioners' challenges, mitigation strategies, and the foreseen implications on data quality. 44% of respondents reported observing LLM usage in their crowdsourced data. While 93% of them had anticipated this, half were unsure what precautions to take. The most prevalent detection strategies are distinctive textual style patterns and unusually fast completion times. Overall, survey responses show that the research community is aware of the problem and taking measures, but existing efforts remain insufficient to fully address it. Finally, we derive a set of considerations to guide future crowdsourced free-text data collection in the era of LLMs.

Related papers

• A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents [1.6643984790651822]
  Large language models are increasingly used by participants on crowdsourcing platforms when responding to surveys.<n>Our study aims to quantify the prevalence of this behavior and investigate methods to detect and prevent it.
  arXiv  Detail & Related papers  (2026-07-01T04:01:26Z)
• CHASM: Unveiling Covert Advertisements on Chinese Social Media [14.328587848128201]
  We present the CHASM, a first-of-its-kind dataset designed to evaluate the capability of Multimodal Large Language Models (MLLMs) in detecting covert advertisements on social media.<n> CHASM is a high-quality, anonymized, manually curated dataset consisting of 4,992 instances, based on real-world scenarios from the Chinese social media platform Rednote.<n>The results show that under both zero-shot and in-context learning settings, none of the current MLLMs are sufficiently reliable for detecting covert advertisements.
  arXiv  Detail & Related papers  (2026-04-22T12:53:36Z)
• Teaching Language Models To Gather Information Proactively [53.85419549904644]
  Large language models (LLMs) are increasingly expected to function as collaborative partners.<n>In this work, we introduce a new task paradigm: proactive information gathering.<n>We design a scalable framework that generates partially specified, real-world tasks, masking key information.<n>Within this setup, our core innovation is a reinforcement finetuning strategy that rewards questions that elicit genuinely new, implicit user information.
  arXiv  Detail & Related papers  (2025-07-28T23:50:09Z)
• Search Arena: Analyzing Search-Augmented LLMs [61.28673331156436]
  We introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions.<n>The dataset spans diverse intents and languages, and contains full system traces with around 12,000 human preference votes.<n>Our analysis reveals that user preferences are influenced by the number of citations, even when the cited content does not directly support the attributed claims.
  arXiv  Detail & Related papers  (2025-06-05T17:59:26Z)
• Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models [52.439289085318634]
  We show how to identify training data known to proprietary large language models (LLMs) by using information-guided probes.<n>Our work builds on a key observation: text passages with high surprisal are good search material for memorization probes.
  arXiv  Detail & Related papers  (2025-03-15T10:19:15Z)
• NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews [65.35458530702442]
  We focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN. LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions.
  arXiv  Detail & Related papers  (2024-11-21T01:37:38Z)
• A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions [0.0]
  Large Language Models (LLMs) have revolutionized various applications in natural language processing (NLP) by providing unprecedented text generation, translation, and comprehension capabilities. Their widespread deployment has brought to light significant concerns regarding biases embedded within these models. This paper presents a comprehensive survey of biases in LLMs, aiming to provide an extensive review of the types, sources, impacts, and mitigation strategies related to these biases.
  arXiv  Detail & Related papers  (2024-09-24T19:50:38Z)
• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
  Large language models (LLMs) have shown remarkable versatility in various generative tasks. This study focuses on the topic of LLMs assist NLP Researchers. To our knowledge, this is the first work to provide such a comprehensive analysis.
  arXiv  Detail & Related papers  (2024-06-24T01:30:22Z)
• Factuality of Large Language Models: A Survey [29.557596701431827]
  We critically analyze existing work with the aim to identify the major challenges and their associated causes. We analyze the obstacles to automated factuality evaluation for open-ended text generation.
  arXiv  Detail & Related papers  (2024-02-04T09:36:31Z)
• Aligning Large Language Models with Human: A Survey [53.6014921995006]
  Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect information. This survey presents a comprehensive overview of these alignment technologies, including the following aspects.
  arXiv  Detail & Related papers  (2023-07-24T17:44:58Z)
• What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? [31.39009622826369]
  We compare the efficacy of interventions that have been proposed in prior work as ways of improving data quality. We find that asking workers to write explanations for their examples is an ineffective stand-alone strategy for boosting NLU example difficulty. We observe that the data from the iterative protocol with expert assessments is more challenging by several measures.
  arXiv  Detail & Related papers  (2021-06-01T21:05:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.