Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?

• URL: http://arxiv.org/abs/2204.13065v1
• Date: Tue, 26 Apr 2022 05:15:58 GMT
• Title: Treating Crowdsourcing as Examination: How to Score Tasks and Online Workers?
• Authors: Guangyang Han and Sufang Li and Runmin Wang and Chunming Wu
• Abstract summary: We try to model workers as four types based on their ability: expert, normal worker, sloppy worker and spammer. We score workers' ability mainly on the medium difficult tasks, then reducing the weight of answers from sloppy workers and modifying the answers from spammers.
• Score: 7.403065976821757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Crowdsourcing is an online outsourcing mode which can solve the current machine learning algorithm's urge need for massive labeled data. Requester posts tasks on crowdsourcing platforms, which employ online workers over the Internet to complete tasks, then aggregate and return results to requester. How to model the interaction between different types of workers and tasks is a hot spot. In this paper, we try to model workers as four types based on their ability: expert, normal worker, sloppy worker and spammer, and divide tasks into hard, medium and easy task according to their difficulty. We believe that even experts struggle with difficult tasks while sloppy workers can get easy tasks right, and spammers always give out wrong answers deliberately. So, good examination tasks should have moderate degree of difficulty and discriminability to score workers more objectively. Thus, we first score workers' ability mainly on the medium difficult tasks, then reducing the weight of answers from sloppy workers and modifying the answers from spammers when inferring the tasks' ground truth. A probability graph model is adopted to simulate the task execution process, and an iterative method is adopted to calculate and update the ground truth, the ability of workers and the difficulty of the task successively. We verify the rightness and effectiveness of our algorithm both in simulated and real crowdsourcing scenes.

Related papers

• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets [73.2096288987301]
  We propose a simple approach that uses a small amount of downstream expert data to selectively query relevant behaviors from an offline, unlabeled dataset. We observe that our method learns to query only the relevant transitions to the task, filtering out sub-optimal or task-irrelevant data. Our simple querying approach outperforms more complex goal-conditioned methods by 20% across simulated and real robotic manipulation tasks from images.
  arXiv  Detail & Related papers  (2023-04-18T05:42:53Z)
• Task Discovery: Finding the Tasks that Neural Networks Generalize on [1.4043229953691112]
  We show that one set of images can give rise to many tasks on which neural networks generalize well. As an example, we show that the discovered tasks can be used to automatically create adversarial train-test splits.
  arXiv  Detail & Related papers  (2022-12-01T03:57:48Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Divide & Conquer Imitation Learning [75.31752559017978]
  Imitation Learning can be a powerful approach to bootstrap the learning process. We present a novel algorithm designed to imitate complex robotic tasks from the states of an expert trajectory. We show that our method imitates a non-holonomic navigation task and scales to a complex simulated robotic manipulation task with very high sample efficiency.
  arXiv  Detail & Related papers  (2022-04-15T09:56:50Z)
• A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental Limits [20.955889997204693]
  Crowdsourcing system has emerged as an effective platform for labeling data with relatively low cost by using non-expert workers. In this paper, we consider a new model, called $d$-type specialization model, in which each task and worker has its own (unknown) type. We propose label inference algorithms achieving the order-wise optimal limit even when the types of tasks or those of workers are unknown.
  arXiv  Detail & Related papers  (2021-11-19T05:32:59Z)
• Efficiently Identifying Task Groupings for Multi-Task Learning [55.80489920205404]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single training run by co-training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss.
  arXiv  Detail & Related papers  (2021-09-10T02:01:43Z)
• What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? [31.39009622826369]
  We compare the efficacy of interventions that have been proposed in prior work as ways of improving data quality. We find that asking workers to write explanations for their examples is an ineffective stand-alone strategy for boosting NLU example difficulty. We observe that the data from the iterative protocol with expert assessments is more challenging by several measures.
  arXiv  Detail & Related papers  (2021-06-01T21:05:52Z)
• Adaptive Task Sampling for Meta-Learning [79.61146834134459]
  Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time. We propose an adaptive task sampling method to improve the generalization performance.
  arXiv  Detail & Related papers  (2020-07-17T03:15:53Z)
• Algorithms for Hiring and Outsourcing in the Online Labor Market [12.893230873578878]
  We provide algorithms for outsourcing and hiring workers in a general setting. We call this model team formation with outsourcing. Our contribution is an efficient online cost-minimizing algorithm for hiring and firing team members and outsourcing tasks.
  arXiv  Detail & Related papers  (2020-02-16T18:56:26Z)
• Algorithms for Fair Team Formation in Online Labour Marketplaces [6.446179861303341]
  We aim to guarantee that the process of hiring workers through the use of machine learning and algorithmic data analysis tools does not discriminate, even unintentionally, on grounds of nationality or gender. We provide inapproximability results for the Fair Team Formation problem together with four algorithms for the problem itself. We also tested the effectiveness of our algorithmic solutions by performing experiments using real data from an online labor marketplace.
  arXiv  Detail & Related papers  (2020-02-14T11:33:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.