Crowdsourcing with Meta-Workers: A New Way to Save the Budget
- URL: http://arxiv.org/abs/2111.04068v1
- Date: Sun, 7 Nov 2021 12:40:29 GMT
- Title: Crowdsourcing with Meta-Workers: A New Way to Save the Budget
- Authors: Guangyang Han, Guoxian Yu, Lizhen Cui, Carlotta Domeniconi, Xiangliang
Zhang
- Abstract summary: We introduce the concept of emphmeta-worker, a machine annotator trained by meta learning for types of tasks that are well-fit for AI.
Unlike regular crowd workers, meta-workers can be reliable, stable, and more importantly, tireless and free.
- Score: 50.04836252733443
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Due to the unreliability of Internet workers, it's difficult to complete a
crowdsourcing project satisfactorily, especially when the tasks are multiple
and the budget is limited. Recently, meta learning has brought new vitality to
few-shot learning, making it possible to obtain a classifier with a fair
performance using only a few training samples. Here we introduce the concept of
\emph{meta-worker}, a machine annotator trained by meta learning for types of
tasks (i.e., image classification) that are well-fit for AI. Unlike regular
crowd workers, meta-workers can be reliable, stable, and more importantly,
tireless and free. We first cluster unlabeled data and ask crowd workers to
repeatedly annotate the instances nearby the cluster centers; we then leverage
the annotated data and meta-training datasets to build a cluster of
meta-workers using different meta learning algorithms. Subsequently,
meta-workers are asked to annotate the remaining crowdsourced tasks. The
Jensen-Shannon divergence is used to measure the disagreement among the
annotations provided by the meta-workers, which determines whether or not crowd
workers should be invited for further annotation of the same task. Finally, we
model meta-workers' preferences and compute the consensus annotation by
weighted majority voting. Our empirical study confirms that, by combining
machine and human intelligence, we can accomplish a crowdsourcing project with
a lower budget than state-of-the-art task assignment methods, while achieving a
superior or comparable quality.
Related papers
- Improve Meta-learning for Few-Shot Text Classification with All You Can Acquire from the Tasks [10.556477506959888]
Existing methods often encounter difficulties in drawing accurate class prototypes from support set samples.
Recent approaches attempt to incorporate external knowledge or pre-trained language models to augment data, but this requires additional resources.
We propose a novel solution by adequately leveraging the information within the task itself.
arXiv Detail & Related papers (2024-10-14T12:47:11Z) - Meta Knowledge Condensation for Federated Learning [65.20774786251683]
Existing federated learning paradigms usually extensively exchange distributed models at a central solver to achieve a more powerful model.
This would incur severe communication burden between a server and multiple clients especially when data distributions are heterogeneous.
Unlike existing paradigms, we introduce an alternative perspective to significantly decrease the communication cost in federate learning.
arXiv Detail & Related papers (2022-09-29T15:07:37Z) - Does MAML Only Work via Feature Re-use? A Data Centric Perspective [19.556093984142418]
We provide empirical results that shed some light on how meta-learned MAML representations function.
We show that it is possible to define a family of synthetic benchmarks that result in a low degree of feature re-use.
We conjecture the core challenge of re-thinking meta-learning is in the design of few-shot learning data sets and benchmarks.
arXiv Detail & Related papers (2021-12-24T20:18:38Z) - Meta-Regularization by Enforcing Mutual-Exclusiveness [0.8057006406834467]
We propose a regularization technique for meta-learning models that gives the model designer more control over the information flow during meta-training.
Our proposed regularization function shows an accuracy boost of $sim$ $36%$ on the Omniglot dataset.
arXiv Detail & Related papers (2021-01-24T22:57:19Z) - COG: Connecting New Skills to Past Experience with Offline Reinforcement
Learning [78.13740204156858]
We show that we can reuse prior data to extend new skills simply through dynamic programming.
We demonstrate the effectiveness of our approach by chaining together several behaviors seen in prior datasets for solving a new task.
We train our policies in an end-to-end fashion, mapping high-dimensional image observations to low-level robot control commands.
arXiv Detail & Related papers (2020-10-27T17:57:29Z) - Self-Supervised Meta-Learning for Few-Shot Natural Language
Classification Tasks [40.97125791174191]
We propose a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text.
We show that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning.
arXiv Detail & Related papers (2020-09-17T17:53:59Z) - Improving Generalization in Meta-learning via Task Augmentation [69.83677015207527]
We propose two task augmentation methods, including MetaMix and Channel Shuffle.
Both MetaMix and Channel Shuffle outperform state-of-the-art results by a large margin across many datasets.
arXiv Detail & Related papers (2020-07-26T01:50:42Z) - Adaptive Task Sampling for Meta-Learning [79.61146834134459]
Key idea of meta-learning for few-shot classification is to mimic the few-shot situations faced at test time.
We propose an adaptive task sampling method to improve the generalization performance.
arXiv Detail & Related papers (2020-07-17T03:15:53Z) - Incremental Meta-Learning via Indirect Discriminant Alignment [118.61152684795178]
We develop a notion of incremental learning during the meta-training phase of meta-learning.
Our approach performs favorably at test time as compared to training a model with the full meta-training set.
arXiv Detail & Related papers (2020-02-11T01:39:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.