LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems

• URL: http://arxiv.org/abs/2403.09810v1
• Date: Thu, 14 Mar 2024 18:59:10 GMT
• Title: LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems
• Authors: Chu Li, Zhihan Zhang, Michael Saugstad, Esteban Safranchik, Minchu Kulkarni, Xiaoyu Huang, Shwetak Patel, Vikram Iyer, Tim Althoff, Jon E. Froehlich,
• Abstract summary: This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility.
• Score: 16.546017147593044
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Crowdsourcing platforms have transformed distributed problem-solving, yet quality control remains a persistent challenge. Traditional quality control measures, such as prescreening workers and refining instructions, often focus solely on optimizing economic output. This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness based on user behavior and domain knowledge. Our technical evaluation shows that our LabelAId pipeline consistently outperforms state-of-the-art ML baselines, improving mistake inference accuracy by 36.7% with 50 downstream samples. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility. A between-subjects study with 34 participants demonstrates that LabelAId significantly enhances label precision without compromising efficiency while also increasing labeler confidence. We discuss LabelAId's success factors, limitations, and its generalizability to other crowdsourced science domains.

Related papers

• Quality Sentinel: Estimating Label Quality and Errors in Medical Segmentation Datasets [11.134987228105162]
  We introduce a regression model, Quality Sentinel, to estimate label quality compared with manual annotations in medical segmentation datasets. This regression model was trained on over 4 million image-label pairs created by us. Our Quality Sentinel can predict the label quality of 142 body structures.
  arXiv  Detail & Related papers  (2024-06-01T07:03:15Z)
• Crowdsourcing with Enhanced Data Quality Assurance: An Efficient Approach to Mitigate Resource Scarcity Challenges in Training Large Language Models for Healthcare [0.0]
  We propose a crowdsourcing framework enriched with quality control measures at the pre-, real-time, and post-data gathering stages. Our study evaluated the effectiveness of enhancing data quality through its impact on Large Language Models for predicting autism-related symptoms.
  arXiv  Detail & Related papers  (2024-05-16T08:29:00Z)
• Soften to Defend: Towards Adversarial Robustness via Self-Guided Label Refinement [5.865750284677784]
  Adversarial training (AT) is one of the most effective ways to obtain the robustness of deep neural networks against adversarial attacks. AT methods suffer from robust overfitting, i.e., a significant generalization gap between the training and testing curves. We propose a label refinement approach for AT, which self-refines a more accurate and informative label distribution from over-confident hard labels.
  arXiv  Detail & Related papers  (2024-03-14T04:48:31Z)
• H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognition [0.9298134918423911]
  We show that correcting 5% of labels can close the AIhuman performance gap by up to 64% relative improvement. We also show that correcting 20% of labels can close the performance gap by up to 86% relative improvement.
  arXiv  Detail & Related papers  (2023-11-20T18:16:27Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• A Weakly Supervised Learning Framework for Salient Object Detection via Hybrid Labels [96.56299163691979]
  This paper focuses on a new weakly-supervised salient object detection (SOD) task under hybrid labels. To address the issues of label noise and quantity imbalance in this task, we design a new pipeline framework with three sophisticated training strategies. Experiments on five SOD benchmarks show that our method achieves competitive performance against weakly-supervised/unsupervised methods.
  arXiv  Detail & Related papers  (2022-09-07T06:45:39Z)
• Adapting to Online Label Shift with Provable Guarantees [137.89382409682233]
  We formulate and investigate the problem of online label shift. The non-stationarity and lack of supervision make the problem challenging to be tackled. Our algorithms enjoy optimal dynamic regret, indicating that performance is competitive with a clairvoyant nature.
  arXiv  Detail & Related papers  (2022-07-05T15:43:14Z)
• Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing [0.0]
  We show that fidelity of the ground truth data can lead to spurious differences in performance of ASRs between urban and rural populations. Our findings highlight how trade-offs between label quality and data annotation costs can complicate algorithmic audits in practice.
  arXiv  Detail & Related papers  (2021-11-16T19:00:03Z)
• Creating Training Sets via Weak Indirect Supervision [66.77795318313372]
  Weak Supervision (WS) frameworks synthesize training labels from multiple potentially noisy supervision sources. We formulate Weak Indirect Supervision (WIS), a new research problem for automatically synthesizing training labels. We develop a probabilistic modeling approach, PLRM, which uses user-provided label relations to model and leverage indirect supervision sources.
  arXiv  Detail & Related papers  (2021-10-07T14:09:35Z)
• Mutual Mean-Teaching: Pseudo Label Refinery for Unsupervised Domain Adaptation on Person Re-identification [56.97651712118167]
  Person re-identification (re-ID) aims at identifying the same persons' images across different cameras. domain diversities between different datasets pose an evident challenge for adapting the re-ID model trained on one dataset to another one. We propose an unsupervised framework, Mutual Mean-Teaching (MMT), to learn better features from the target domain via off-line refined hard pseudo labels and on-line refined soft pseudo labels.
  arXiv  Detail & Related papers  (2020-01-06T12:42:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.