Label Critic: Design Data Before Models

• URL: http://arxiv.org/abs/2411.02753v1
• Date: Tue, 05 Nov 2024 02:50:47 GMT
• Title: Label Critic: Design Data Before Models
• Authors: Pedro R. A. S. Bassi, Qilong Wu, Wenxuan Li, Sergio Decherchi, Andrea Cavalli, Alan Yuille, Zongwei Zhou,
• Abstract summary: We develop an automatic tool, called Label Critic, that can assess label quality through pairwise comparisons. When the Best-AI Labels are sufficiently accurate (81% depending on body structures), they will be directly adopted as the gold-standard annotations for the dataset. Label Critic can also check the label quality of a single AI Label with 71.8% accuracy when no alternatives are available for comparison.
• Score: 7.142066343369597
• License:
• Abstract: As medical datasets rapidly expand, creating detailed annotations of different body structures becomes increasingly expensive and time-consuming. We consider that requesting radiologists to create detailed annotations is unnecessarily burdensome and that pre-existing AI models can largely automate this process. Following the spirit don't use a sledgehammer on a nut, we find that, rather than creating annotations from scratch, radiologists only have to review and edit errors if the Best-AI Labels have mistakes. To obtain the Best-AI Labels among multiple AI Labels, we developed an automatic tool, called Label Critic, that can assess label quality through tireless pairwise comparisons. Extensive experiments demonstrate that, when incorporated with our developed Image-Prompt pairs, pre-existing Large Vision-Language Models (LVLM), trained on natural images and texts, achieve 96.5% accuracy when choosing the best label in a pair-wise comparison, without extra fine-tuning. By transforming the manual annotation task (30-60 min/scan) into an automatic comparison task (15 sec/scan), we effectively reduce the manual efforts required from radiologists by an order of magnitude. When the Best-AI Labels are sufficiently accurate (81% depending on body structures), they will be directly adopted as the gold-standard annotations for the dataset, with lower-quality AI Labels automatically discarded. Label Critic can also check the label quality of a single AI Label with 71.8% accuracy when no alternatives are available for comparison, prompting radiologists to review and edit if the estimated quality is low (19% depending on body structures).

Related papers

• Automatic Labels are as Effective as Manual Labels in Biomedical Images Classification with Deep Learning [32.71326861917332]
  One of the main limitations to train deep learning algorithms to perform a specific task is the need for medical experts to label data. This paper aims to investigate under which circumstances automatic labels can be adopted to train a DL model on the classification of Whole Slide Images (WSI) The application of the Semantic Knowledge Extractor Tool (SKET) algorithm to generate automatic labels leads to performance comparable to the one obtained with manual labels.
  arXiv  Detail & Related papers  (2024-06-20T14:20:50Z)
• LABELMAKER: Automatic Semantic Label Generation from RGB-D Trajectories [59.14011485494713]
  This work introduces a fully automated 2D/3D labeling framework that can generate labels for RGB-D scans at equal (or better) level of accuracy. We demonstrate the effectiveness of our LabelMaker pipeline by generating significantly better labels for the ScanNet datasets and automatically labelling the previously unlabeled ARKitScenes dataset.
  arXiv  Detail & Related papers  (2023-11-20T20:40:24Z)
• H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognition [0.9298134918423911]
  We show that correcting 5% of labels can close the AIhuman performance gap by up to 64% relative improvement. We also show that correcting 20% of labels can close the performance gap by up to 86% relative improvement.
  arXiv  Detail & Related papers  (2023-11-20T18:16:27Z)
• Estimating label quality and errors in semantic segmentation data via any model [19.84626033109009]
  We study methods to score label quality, such that the images with the lowest scores are least likely to be correctly labeled. This helps prioritize what data to review in order to ensure a high-quality training/evaluation dataset.
  arXiv  Detail & Related papers  (2023-07-11T07:29:09Z)
• Automated Labeling of German Chest X-Ray Radiology Reports using Deep Learning [50.591267188664666]
  We propose a deep learning-based CheXpert label prediction model, pre-trained on reports labeled by a rule-based German CheXpert model. Our results demonstrate the effectiveness of our approach, which significantly outperformed the rule-based model on all three tasks.
  arXiv  Detail & Related papers  (2023-06-09T16:08:35Z)
• Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels [61.97359362447732]
  Learning from noisy labels is an important and long-standing problem in machine learning for real applications. In this paper, we reformulate the label-noise problem from a generative-model perspective. Our model achieves new state-of-the-art (SOTA) results on all the standard real-world benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-31T03:01:36Z)
• Transfer learning with weak labels from radiology reports: application to glioma change detection [0.2010294990327175]
  We propose a combined use of weak labels (imprecise, but fast-to-create annotations) and Transfer Learning (TL) Specifically, we explore inductive TL, where source and target domains are identical, but tasks are different due to a label shift. We investigate the relationship between model size and TL, comparing a low-capacity VGG with a higher-capacity SEResNeXt.
  arXiv  Detail & Related papers  (2022-10-18T09:15:27Z)
• TagLab: A human-centric AI system for interactive semantic segmentation [63.84619323110687]
  TagLab is an open-source AI-assisted software for annotating large orthoimages. It speeds up image annotation from scratch through assisted tools, creates custom fully automatic semantic segmentation models, and allows the quick edits of automatic predictions. We report our results in two different scenarios, marine ecology, and architectural heritage.
  arXiv  Detail & Related papers  (2021-12-23T16:50:06Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• Active Learning for Noisy Data Streams Using Weak and Strong Labelers [3.9370369973510746]
  We consider a novel weak and strong labeler problem inspired by humans natural ability for labeling. We propose an on-line active learning algorithm that consists of four steps: filtering, adding diversity, informative sample selection, and labeler selection. We derive a decision function that measures the information gain by combining the informativeness of individual samples and model confidence.
  arXiv  Detail & Related papers  (2020-10-27T09:18:35Z)
• Learning to Purify Noisy Labels via Meta Soft Label Corrector [49.92310583232323]
  Recent deep neural networks (DNNs) can easily overfit to biased training data with noisy labels. Label correction strategy is commonly used to alleviate this issue. We propose a meta-learning model which could estimate soft labels through meta-gradient descent step.
  arXiv  Detail & Related papers  (2020-08-03T03:25:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.