Spectral Clustering for Crowdsourcing with Inherently Distinct Task Types

• URL: http://arxiv.org/abs/2302.07393v2
• Date: Sat, 10 Aug 2024 00:48:05 GMT
• Title: Spectral Clustering for Crowdsourcing with Inherently Distinct Task Types
• Authors: Saptarshi Mandal, Seo Taek Kong, Dimitrios Katselis, R. Srikant,
• Abstract summary: The Dawid-Skene model is the most widely assumed model in the analysis of crowdsourcing algorithms. We show that different weights for different types are necessary for a multi-type model. Numerical experiments show how clustering tasks by type before estimating ground-truth labels enhances the performance of crowdsourcing algorithms.
• Score: 7.788574428889243
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Dawid-Skene model is the most widely assumed model in the analysis of crowdsourcing algorithms that estimate ground-truth labels from noisy worker responses. In this work, we are motivated by crowdsourcing applications where workers have distinct skill sets and their accuracy additionally depends on a task's type. While weighted majority vote (WMV) with a single weight vector for each worker achieves the optimal label estimation error in the Dawid-Skene model, we show that different weights for different types are necessary for a multi-type model. Focusing on the case where there are two types of tasks, we propose a spectral method to partition tasks into two groups that cluster tasks by type. Our analysis reveals that task types can be perfectly recovered if the number of workers $n$ scales logarithmically with the number of tasks $d$. Any algorithm designed for the Dawid-Skene model can then be applied independently to each type to infer the labels. Numerical experiments show how clustering tasks by type before estimating ground-truth labels enhances the performance of crowdsourcing algorithms in practical applications.

Related papers

• A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks [51.14185612418977]
  A strategy to improve label quality is to ask multiple annotators to label the same item and aggregate their labels. While a variety of bespoke models have been proposed for specific tasks, our work is the first to introduce aggregation methods that generalize across many diverse complex tasks. This article extends our prior work with investigation of three new research questions.
  arXiv  Detail & Related papers  (2023-12-20T21:28:35Z)
• Label Selection Approach to Learning from Crowds [25.894399244406287]
  Learning from Crowds is a framework which directly trains the models using noisy labeled data from crowd workers. We propose a novel Learning from Crowds model, inspired by SelectiveNet proposed for the selective prediction problem. A major advantage of the proposed method is that it can be applied to almost all variants of supervised learning problems.
  arXiv  Detail & Related papers  (2023-08-21T00:22:32Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Tyger: Task-Type-Generic Active Learning for Molecular Property Prediction [121.97742787439546]
  How to accurately predict the properties of molecules is an essential problem in AI-driven drug discovery. To reduce annotation cost, deep Active Learning methods are developed to select only the most representative and informative data for annotating. We propose a Task-type-generic active learning framework (termed Tyger) that is able to handle different types of learning tasks in a unified manner.
  arXiv  Detail & Related papers  (2022-05-23T12:56:12Z)
• Estimating Structural Disparities for Face Models [54.062512989859265]
  In machine learning, disparity metrics are often defined by measuring the difference in the performance or outcome of a model, across different sub-populations. We explore performing such analysis on computer vision models trained on human faces, and on tasks such as face attribute prediction and affect estimation.
  arXiv  Detail & Related papers  (2022-04-13T05:30:53Z)
• A Worker-Task Specialization Model for Crowdsourcing: Efficient Inference and Fundamental Limits [20.955889997204693]
  Crowdsourcing system has emerged as an effective platform for labeling data with relatively low cost by using non-expert workers. In this paper, we consider a new model, called $d$-type specialization model, in which each task and worker has its own (unknown) type. We propose label inference algorithms achieving the order-wise optimal limit even when the types of tasks or those of workers are unknown.
  arXiv  Detail & Related papers  (2021-11-19T05:32:59Z)
• Active clustering for labeling training data [0.8029049649310211]
  We propose a setting for training data gathering where the human experts perform the comparatively cheap task of answering pairwise queries. We analyze the algorithms that minimize the average number of queries required to cluster the items and analyze their complexity.
  arXiv  Detail & Related papers  (2021-10-27T15:35:58Z)
• Adaptive Task Sampling for Meta-Learning [79.61146834134459]
  Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time. We propose an adaptive task sampling method to improve the generalization performance.
  arXiv  Detail & Related papers  (2020-07-17T03:15:53Z)
• Variational Bayesian Inference for Crowdsourcing Predictions [6.878219199575748]
  We develop a variational Bayesian technique for two different worker noise models. Our evaluations on synthetic and real-world datasets demonstrate that these approaches perform significantly better than existing non-Bayesian approaches.
  arXiv  Detail & Related papers  (2020-06-01T08:11:50Z)
• Low Resource Multi-Task Sequence Tagging -- Revisiting Dynamic Conditional Random Fields [67.51177964010967]
  We compare different models for low resource multi-task sequence tagging that leverage dependencies between label sequences for different tasks. We find that explicit modeling of inter-dependencies between task predictions outperforms single-task as well as standard multi-task models.
  arXiv  Detail & Related papers  (2020-05-01T07:11:34Z)
• Crowdsourced Labeling for Worker-Task Specialization Model [14.315501760755605]
  We consider crowdsourced labeling under a $d$-type worker-task specialization model. We design an inference algorithm that recovers binary task labels by using worker clustering, worker skill estimation and weighted majority voting.
  arXiv  Detail & Related papers  (2020-03-21T13:27:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.