Study on Patterns and Effect of Task Diversity in Software Crowdsourcing
- URL: http://arxiv.org/abs/2006.00871v2
- Date: Wed, 29 Jul 2020 22:22:36 GMT
- Title: Study on Patterns and Effect of Task Diversity in Software Crowdsourcing
- Authors: Denisse Martinez Mejorado, Razieh Saremi, Ye Yang, and Jose E.
Ramirez-Marquez
- Abstract summary: The objective of this study is to empirically investigate patterns and effect of task diversity in software crowdsourcing platform.
The empirical study is conducted on more than one year's real-world data from TopCoder, the leading software crowdsourcing platform.
- Score: 1.758684872705242
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Context: The success of software crowdsourcing depends on steady tasks supply
and active worker pool. Existing analysis reveals an average task failure ratio
of 15.7% in software crowdsourcing market. Goal: The objective of this study is
to empirically investigate patterns and effect of task diversity in software
crowdsourcing platform in order to improve the success and efficiency of
software crowdsourcing. Method: We propose a conceptual task diversity model,
and develop an approach to measuring and analyzing task diversity.More
specifically, this includes grouping similar tasks, ranking them based on their
competition level and identifying the dominant attributes that distinguish
among these levels, and then studying the impact of task diversity on task
success and worker performance in crowdsourcing platform. The empirical study
is conducted on more than one year's real-world data from TopCoder, the leading
software crowdsourcing platform. Results: We identified that monetary prize and
task complexity are the dominant attributes that differentiate among different
competition levels. Based on these dominant attributes, we found three task
diversity patterns (configurations) from workers behavior perspective:
responsive to prize, responsive to prize and complexity and over responsive to
prize. This study supports that1) responsive to prize configuration provides
highest level of task density and workers' reliability in a platform; 2)
responsive to prize and complexity configuration leads to attracting high level
of trustworthy workers; 3) over responsive to prize configuration results in
highest task stability and the lowest failure ratio in the platform for not
high similar tasks.
Related papers
- WESE: Weak Exploration to Strong Exploitation for LLM Agents [95.6720931773781]
This paper proposes a novel approach, Weak Exploration to Strong Exploitation (WESE) to enhance LLM agents in solving open-world interactive tasks.
WESE involves decoupling the exploration and exploitation process, employing a cost-effective weak agent to perform exploration tasks for global knowledge.
A knowledge graph-based strategy is then introduced to store the acquired knowledge and extract task-relevant knowledge, enhancing the stronger agent in success rate and efficiency for the exploitation task.
arXiv Detail & Related papers (2024-04-11T03:31:54Z) - Evaluating Collaborative and Autonomous Agents in Data-Stream-Supported
Coordination of Mobile Crowdsourcing [0.7865191493201839]
In mobile crowdsourcing, tasks often get assigned to crowdworkers who struggle to complete those tasks successfully.
We propose different mechanisms to achieve outcome prediction and task coordination in mobile crowdsourcing.
arXiv Detail & Related papers (2024-01-23T16:00:45Z) - Diversity-Based Recruitment in Crowdsensing By Combinatorial Multi-Armed
Bandits [6.802315212233411]
This paper explores mobile crowdsensing, which leverages mobile devices and their users for collective sensing tasks under the coordination of a central requester.
The primary challenge here is the variability in the sensing capabilities of individual workers, which are initially unknown and must be progressively learned.
We propose a novel model that enhances task diversity over the rounds by dynamically adjusting the weight of tasks in each round based on their frequency of assignment.
arXiv Detail & Related papers (2023-12-25T13:54:58Z) - Benchmarking Robustness and Generalization in Multi-Agent Systems: A
Case Study on Neural MMO [50.58083807719749]
We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions.
This competition targets robustness and generalization in multi-agent systems.
We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.
arXiv Detail & Related papers (2023-08-30T07:16:11Z) - Towards Task Sampler Learning for Meta-Learning [37.02030832662183]
Meta-learning aims to learn general knowledge with diverse training tasks conducted from limited data, and then transfer it to new tasks.
It is commonly believed that increasing task diversity will enhance the generalization ability of meta-learning models.
This paper challenges this view through empirical and theoretical analysis.
arXiv Detail & Related papers (2023-07-18T01:53:18Z) - Exposing and Addressing Cross-Task Inconsistency in Unified
Vision-Language Models [80.23791222509644]
Inconsistent AI models are considered brittle and untrustworthy by human users.
We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks.
We propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets.
arXiv Detail & Related papers (2023-03-28T16:57:12Z) - Reinforcement Learning with Success Induced Task Prioritization [68.8204255655161]
We introduce Success Induced Task Prioritization (SITP), a framework for automatic curriculum learning.
The algorithm selects the order of tasks that provide the fastest learning for agents.
We demonstrate that SITP matches or surpasses the results of other curriculum design methods.
arXiv Detail & Related papers (2022-12-30T12:32:43Z) - Uni-Perceiver: Pre-training Unified Architecture for Generic Perception
for Zero-shot and Few-shot Tasks [73.63892022944198]
We present a generic perception architecture named Uni-Perceiver.
It processes a variety of modalities and tasks with unified modeling and shared parameters.
Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks.
arXiv Detail & Related papers (2021-12-02T18:59:50Z) - An Evolutionary Algorithm for Task Scheduling in Crowdsourced Software
Development [10.373891804761376]
This paper proposes an evolutionary algorithm-based task scheduling method for crowdsourced software development.
Experimental results on 4 projects demonstrate that the proposed method has the potential to reduce project duration by a factor of 33-78%.
arXiv Detail & Related papers (2021-07-05T18:07:26Z) - Weakly-supervised Multi-task Learning for Multimodal Affect Recognition [33.7929682119287]
We propose to leverage datasets using weakly-supervised multi-task learning to improve generalization performance.
Specifically, we explore three multimodal affect recognition tasks: 1) emotion recognition; 2) sentiment analysis; and 3) sarcasm recognition.
Our experimental results show that multi-tasking can benefit all these tasks, achieving an improvement up to 2.9% accuracy and 3.3% F1-score.
arXiv Detail & Related papers (2021-04-23T12:36:19Z) - Randomized Entity-wise Factorization for Multi-Agent Reinforcement
Learning [59.62721526353915]
Multi-agent settings in the real world often involve tasks with varying types and quantities of agents and non-agent entities.
Our method aims to leverage these commonalities by asking the question: What is the expected utility of each agent when only considering a randomly selected sub-group of its observed entities?''
arXiv Detail & Related papers (2020-06-07T18:28:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.