Redefining Research Crowdsourcing: Incorporating Human Feedback with LLM-Powered Digital Twins

• URL: http://arxiv.org/abs/2505.24004v1
• Date: Thu, 29 May 2025 20:55:27 GMT
• Title: Redefining Research Crowdsourcing: Incorporating Human Feedback with LLM-Powered Digital Twins
• Abstract summary: Crowd work platforms like Amazon Mechanical Turk and Prolific are vital for research, yet workers' growing use of generative AI tools poses challenges.<n>We propose a hybrid framework using digital twins, personalized AI models that emulate workers' behaviors and preferences while keeping humans in the loop.
• Score: 7.900453156741494
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Crowd work platforms like Amazon Mechanical Turk and Prolific are vital for research, yet workers' growing use of generative AI tools poses challenges. Researchers face compromised data validity as AI responses replace authentic human behavior, while workers risk diminished roles as AI automates tasks. To address this, we propose a hybrid framework using digital twins, personalized AI models that emulate workers' behaviors and preferences while keeping humans in the loop. We evaluate our system with an experiment (n=88 crowd workers) and in-depth interviews with crowd workers (n=5) and social science researchers (n=4). Our results suggest that digital twins may enhance productivity and reduce decision fatigue while maintaining response quality. Both researchers and workers emphasized the importance of transparency, ethical data use, and worker agency. By automating repetitive tasks and preserving human engagement for nuanced ones, digital twins may help balance scalability with authenticity.

Related papers

• CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks [55.84031315625504]
  CollabSkill is a framework for evaluating human-agent collaboration on real-world occupational tasks.<n>It pairs real human workers with AI agents on tasks matched to their occupational background.<n>Our analysis reveals that practical experience emerges as the primary driver of collaboration skill.
  arXiv  Detail & Related papers  (2026-04-20T03:03:25Z)
• What's the next frontier for Data-centric AI? Data Savvy Agents [71.76058707995398]
  We argue that data-savvy capabilities should be a top priority in the design of agentic systems.<n>We propose four key capabilities to realize this vision: Proactive data acquisition, Sophisticated data processing, Interactive test data synthesis, and Continual adaptation.
  arXiv  Detail & Related papers  (2025-11-02T17:09:29Z)
• HHI-Assist: A Dataset and Benchmark of Human-Human Interaction in Physical Assistance Scenario [63.77482302352545]
  HHI-Assist is a dataset comprising motion capture clips of human-human interactions in assistive tasks.<n>Our work has the potential to significantly enhance robotic assistance policies.
  arXiv  Detail & Related papers  (2025-09-12T09:38:17Z)
• Vibe Coding: Is Human Nature the Ghost in the Machine? [0.0]
  We analyzed three "vibe coding" sessions between a human product lead and an AI software engineer.<n>We investigated similarities and differences in team dynamics, communication patterns, and development outcomes.<n>To our surprise, later conversations revealed that the AI agent had systematically misrepresented its accomplishments.
  arXiv  Detail & Related papers  (2025-08-28T15:48:48Z)
• A roadmap for AI in robotics [55.87087746398059]
  We are witnessing growing excitement in robotics at the prospect of leveraging the potential of AI to tackle some of the outstanding barriers to the full deployment of robots in our daily lives.<n>This article offers an assessment of what AI for robotics has achieved since the 1990s and proposes a short- and medium-term research roadmap listing challenges and promises.
  arXiv  Detail & Related papers  (2025-07-26T15:18:28Z)
• AI, Humans, and Data Science: Optimizing Roles Across Workflows and the Workforce [0.0]
  We consider the potential and limitation of analytic, generative, and agentic AI to augment data scientists or take on tasks traditionally done by human analysts and researchers.<n>Just as earlier eras of survey analysis created issues when the increased ease of using statistical software allowed researchers to conduct analyses they did not fully understand, the new AI tools may create similar but larger risks.
  arXiv  Detail & Related papers  (2025-07-15T17:59:06Z)
• SIME: Enhancing Policy Self-Improvement with Modal-level Exploration [49.86173021151849]
  We identify the key to successful self-improvement: modal-level exploration and data selection.<n>By incorporating a modal-level exploration mechanism during policy execution, the robot can produce more diverse and multi-modal interactions.<n>We successfully demonstrate effective robot self-improvement on both simulation benchmarks and real-world experiments.
  arXiv  Detail & Related papers  (2025-05-02T17:13:03Z)
• AI-Driven Agents with Prompts Designed for High Agreeableness Increase the Likelihood of Being Mistaken for a Human in the Turing Test [0.0]
  GPT agents with varying levels of agreeableness were tested in a Turing Test. All exceeded a 50% confusion rate, with the highly agreeable AI agent surpassing 60%. This agent was also recognized as exhibiting the most human-like traits.
  arXiv  Detail & Related papers  (2024-11-20T23:12:49Z)
• Measuring Human Contribution in AI-Assisted Content Generation [66.06040950325969]
  This study raises the research question of measuring human contribution in AI-assisted content generation.<n>By calculating mutual information between human input and AI-assisted output relative to self-information of AI-assisted output, we quantify the proportional information contribution of humans in content generation.
  arXiv  Detail & Related papers  (2024-08-27T05:56:04Z)
• Towards the Terminator Economy: Assessing Job Exposure to AI through LLMs [10.844598404826355]
  One-third of U.S. employment is highly exposed to AI, primarily in high-skill jobs requiring a graduate or postgraduate level of education.<n>Even in high-skill occupations, AI exhibits high variability in task substitution, suggesting that AI and humans complement each other within the same occupation.<n>All results, models, and code are freely available online to allow the community to reproduce our results, compare outcomes, and use our work as a benchmark to monitor AI's progress over time.
  arXiv  Detail & Related papers  (2024-07-27T08:14:18Z)
• Explainable Human-AI Interaction: A Planning Perspective [32.477369282996385]
  AI systems need to be explainable to the humans in the loop. We will discuss how the AI agent can use mental models to either conform to human expectations, or change those expectations through explanatory communication. While the main focus of the book is on cooperative scenarios, we will point out how the same mental models can be used for obfuscation and deception.
  arXiv  Detail & Related papers  (2024-05-19T22:22:21Z)
• AI and Jobs: Has the Inflection Point Arrived? Evidence from an Online Labor Platform [0.13124513975412255]
  We investigate how AI influences freelancers across different online labor markets (OLMs) To shed light on the underlying mechanisms, we developed a Cournot-type competition model. We find that U.S. web developers tend to benefit more from the release of ChatGPT compared to their counterparts in other regions.
  arXiv  Detail & Related papers  (2023-12-07T10:06:34Z)
• Can AI Serve as a Substitute for Human Subjects in Software Engineering Research? [24.39463126056733]
  This vision paper proposes a novel approach to qualitative data collection in software engineering research by harnessing the capabilities of artificial intelligence (AI) We explore the potential of AI-generated synthetic text as an alternative source of qualitative data. We discuss the prospective development of new foundation models aimed at emulating human behavior in observational studies and user evaluations.
  arXiv  Detail & Related papers  (2023-11-18T14:05:52Z)
• Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots [119.55240471433302]
  Habitat 3.0 is a simulation platform for studying collaborative human-robot tasks in home environments. It addresses challenges in modeling complex deformable bodies and diversity in appearance and motion. Human-in-the-loop infrastructure enables real human interaction with simulated robots via mouse/keyboard or a VR interface.
  arXiv  Detail & Related papers  (2023-10-19T17:29:17Z)
• Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling [73.14508303965683]
  We interviewed eighteen data workers from both industry and academia to learn where and how they would like to collaborate with AI.<n>Surprisingly, though the participants showed excitement about collaborating with AI, many of them also expressed reluctance and pointed out nuanced reasons.
  arXiv  Detail & Related papers  (2023-04-17T15:30:05Z)
• Joint Mind Modeling for Explanation Generation in Complex Human-Robot Collaborative Tasks [83.37025218216888]
  We propose a novel explainable AI (XAI) framework for achieving human-like communication in human-robot collaborations. The robot builds a hierarchical mind model of the human user and generates explanations of its own mind as a form of communications. Results show that the generated explanations of our approach significantly improves the collaboration performance and user perception of the robot.
  arXiv  Detail & Related papers  (2020-07-24T23:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.