H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognition

• URL: http://arxiv.org/abs/2311.11981v1
• Date: Mon, 20 Nov 2023 18:16:27 GMT
• Title: H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognition
• Authors: Xiaojing Duan, John P. Lalor
• Abstract summary: We show that correcting 5% of labels can close the AIhuman performance gap by up to 64% relative improvement. We also show that correcting 20% of labels can close the performance gap by up to 86% relative improvement.
• Score: 0.9298134918423911
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the rapid advancement of machine learning models for NLP tasks, collecting high-fidelity labels from AI models is a realistic possibility. Firms now make AI available to customers via predictions as a service (PaaS). This includes PaaS products for healthcare. It is unclear whether these labels can be used for training a local model without expensive annotation checking by in-house experts. In this work, we propose a new framework for Human Correction of AI-Generated Labels (H-COAL). By ranking AI-generated outputs, one can selectively correct labels and approach gold standard performance (100% human labeling) with significantly less human effort. We show that correcting 5% of labels can close the AI-human performance gap by up to 64% relative improvement, and correcting 20% of labels can close the performance gap by up to 86% relative improvement.

Related papers

• Label Critic: Design Data Before Models [7.142066343369597]
  We develop an automatic tool, called Label Critic, that can assess label quality through pairwise comparisons. When the Best-AI Labels are sufficiently accurate (81% depending on body structures), they will be directly adopted as the gold-standard annotations for the dataset. Label Critic can also check the label quality of a single AI Label with 71.8% accuracy when no alternatives are available for comparison.
  arXiv  Detail & Related papers  (2024-11-05T02:50:47Z)
• LabelAId: Just-in-time AI Interventions for Improving Human Labeling Quality and Domain Knowledge in Crowdsourcing Systems [16.546017147593044]
  This paper explores just-in-time AI interventions to enhance both labeling quality and domain-specific knowledge among crowdworkers. We introduce LabelAId, an advanced inference model combining Programmatic Weak Supervision (PWS) with FT-Transformers to infer label correctness. We then implemented LabelAId into Project Sidewalk, an open-source crowdsourcing platform for urban accessibility.
  arXiv  Detail & Related papers  (2024-03-14T18:59:10Z)
• Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge [18.48059187629883]
  We organized the FLARE 2022 Challenge, the largest abdominal organ analysis challenge to date, to benchmark fast, low-resource, accurate, annotation-efficient, and generalized AI algorithms. We constructed an intercontinental and multinational dataset from more than 50 medical groups, including Computed Tomography (CT) scans with different races, diseases, phases, and manufacturers. Best-performing algorithms successfully generalized to holdout external validation sets, achieving a median DSC of 89.5%, 90.9%, and 88.3% on North American, European, and Asian cohorts, respectively.
  arXiv  Detail & Related papers  (2023-08-10T21:51:48Z)
• Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels [61.97359362447732]
  Learning from noisy labels is an important and long-standing problem in machine learning for real applications. In this paper, we reformulate the label-noise problem from a generative-model perspective. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-31T03:01:36Z)
• Improved Adaptive Algorithm for Scalable Active Learning with Weak Labeler [89.27610526884496]
  Weak Labeler Active Cover (WL-AC) is able to robustly leverage the lower quality weak labelers to reduce the query complexity while retaining the desired level of accuracy. We show its effectiveness on the corrupted-MNIST dataset by significantly reducing the number of labels while keeping the same accuracy as in passive learning.
  arXiv  Detail & Related papers  (2022-11-04T02:52:54Z)
• Eliciting and Learning with Soft Labels from Every Annotator [31.10635260890126]
  We focus on efficiently eliciting soft labels from individual annotators. We demonstrate that learning with our labels achieves comparable model performance to prior approaches.
  arXiv  Detail & Related papers  (2022-07-02T12:03:00Z)
• Going Beyond One-Hot Encoding in Classification: Can Human Uncertainty Improve Model Performance? [14.610038284393166]
  We show that label uncertainty is explicitly embedded into the training process via distributional labels. The incorporation of label uncertainty helps the model to generalize better to unseen data and increases model performance. Similar to existing calibration methods, the distributional labels lead to better-calibrated probabilities, which in turn yield more certain and trustworthy predictions.
  arXiv  Detail & Related papers  (2022-05-30T17:19:11Z)
• Delving into Probabilistic Uncertainty for Unsupervised Domain Adaptive Person Re-Identification [54.174146346387204]
  We propose an approach named probabilistic uncertainty guided progressive label refinery (P$2$LR) for domain adaptive person re-identification. A quantitative criterion is established to measure the uncertainty of pseudo labels and facilitate the network training. Our method outperforms the baseline by 6.5% mAP on the Duke2Market task, while surpassing the state-of-the-art method by 2.5% mAP on the Market2MSMT task.
  arXiv  Detail & Related papers  (2021-12-28T07:40:12Z)
• TagLab: A human-centric AI system for interactive semantic segmentation [63.84619323110687]
  TagLab is an open-source AI-assisted software for annotating large orthoimages. It speeds up image annotation from scratch through assisted tools, creates custom fully automatic semantic segmentation models, and allows the quick edits of automatic predictions. We report our results in two different scenarios, marine ecology, and architectural heritage.
  arXiv  Detail & Related papers  (2021-12-23T16:50:06Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• Learning to Purify Noisy Labels via Meta Soft Label Corrector [49.92310583232323]
  Recent deep neural networks (DNNs) can easily overfit to biased training data with noisy labels. Label correction strategy is commonly used to alleviate this issue. We propose a meta-learning model which could estimate soft labels through meta-gradient descent step.
  arXiv  Detail & Related papers  (2020-08-03T03:25:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.