Crowdsourced Labeling for Worker-Task Specialization Model

• URL: http://arxiv.org/abs/2004.00101v2
• Date: Wed, 9 Jun 2021 06:55:56 GMT
• Title: Crowdsourced Labeling for Worker-Task Specialization Model
• Authors: Doyeon Kim and Hye Won Chung
• Abstract summary: We consider crowdsourced labeling under a $d$-type worker-task specialization model. We design an inference algorithm that recovers binary task labels by using worker clustering, worker skill estimation and weighted majority voting.
• Score: 14.315501760755605
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider crowdsourced labeling under a $d$-type worker-task specialization model, where each worker and task is associated with one particular type among a finite set of types and a worker provides a more reliable answer to tasks of the matched type than to tasks of unmatched types. We design an inference algorithm that recovers binary task labels (up to any given recovery accuracy) by using worker clustering, worker skill estimation and weighted majority voting. The designed inference algorithm does not require any information about worker/task types, and achieves any targeted recovery accuracy with the best known performance (minimum number of queries per task).

Related papers

• Diversity-Based Recruitment in Crowdsensing By Combinatorial Multi-Armed Bandits [6.802315212233411]
  This paper explores mobile crowdsensing, which leverages mobile devices and their users for collective sensing tasks under the coordination of a central requester. The primary challenge here is the variability in the sensing capabilities of individual workers, which are initially unknown and must be progressively learned. We propose a novel model that enhances task diversity over the rounds by dynamically adjusting the weight of tasks in each round based on their frequency of assignment.
  arXiv  Detail & Related papers  (2023-12-25T13:54:58Z)
• A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks [51.14185612418977]
  A strategy to improve label quality is to ask multiple annotators to label the same item and aggregate their labels. While a variety of bespoke models have been proposed for specific tasks, our work is the first to introduce aggregation methods that generalize across many diverse complex tasks. This article extends our prior work with investigation of three new research questions.
  arXiv  Detail & Related papers  (2023-12-20T21:28:35Z)
• Spectral Clustering for Crowdsourcing with Inherently Distinct Task Types [7.788574428889243]
  The Dawid-Skene model is the most widely assumed model in the analysis of crowdsourcing algorithms. We show that different weights for different types are necessary for a multi-type model. Numerical experiments show how clustering tasks by type before estimating ground-truth labels enhances the performance of crowdsourcing algorithms.
  arXiv  Detail & Related papers  (2023-02-14T23:30:39Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Association Graph Learning for Multi-Task Classification with Category Shifts [68.58829338426712]
  We focus on multi-task classification, where related classification tasks share the same label space and are learned simultaneously. We learn an association graph to transfer knowledge among tasks for missing classes. Our method consistently performs better than representative baselines.
  arXiv  Detail & Related papers  (2022-10-10T12:37:41Z)
• Unsupervised Crowdsourcing with Accuracy and Cost Guarantees [4.008789789191313]
  We consider the problem of cost-optimal utilization of a crowdsourcing platform for binary, unsupervised classification of a collection of items. We propose algorithms for acquiring label predictions from workers, and for inferring the true labels of items.
  arXiv  Detail & Related papers  (2022-07-05T12:14:11Z)
• Improving Multi-task Generalization Ability for Neural Text Matching via Prompt Learning [54.66399120084227]
  Recent state-of-the-art neural text matching models (PLMs) are hard to generalize to different tasks. We adopt a specialization-generalization training strategy and refer to it as Match-Prompt. In specialization stage, descriptions of different matching tasks are mapped to only a few prompt tokens. In generalization stage, text matching model explores the essential matching signals by being trained on diverse multiple matching tasks.
  arXiv  Detail & Related papers  (2022-04-06T11:01:08Z)
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks [73.63892022944198]
  We present a generic perception architecture named Uni-Perceiver. It processes a variety of modalities and tasks with unified modeling and shared parameters. Results show that our pre-trained model without any tuning can achieve reasonable performance even on novel tasks.
  arXiv  Detail & Related papers  (2021-12-02T18:59:50Z)
• A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental Limits [20.955889997204693]
  Crowdsourcing system has emerged as an effective platform for labeling data with relatively low cost by using non-expert workers. In this paper, we consider a new model, called $d$-type specialization model, in which each task and worker has its own (unknown) type. We propose label inference algorithms achieving the order-wise optimal limit even when the types of tasks or those of workers are unknown.
  arXiv  Detail & Related papers  (2021-11-19T05:32:59Z)
• Active clustering for labeling training data [0.8029049649310211]
  We propose a setting for training data gathering where the human experts perform the comparatively cheap task of answering pairwise queries. We analyze the algorithms that minimize the average number of queries required to cluster the items and analyze their complexity.
  arXiv  Detail & Related papers  (2021-10-27T15:35:58Z)
• Adaptive Task Sampling for Meta-Learning [79.61146834134459]
  Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time. We propose an adaptive task sampling method to improve the generalization performance.
  arXiv  Detail & Related papers  (2020-07-17T03:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.