How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model

• URL: http://arxiv.org/abs/2312.10600v3
• Date: Thu, 23 Jan 2025 16:02:29 GMT
• Title: How to Efficiently Annotate Images for Best-Performing Deep Learning Based Segmentation Models: An Empirical Study with Weak and Noisy Annotations and Segment Anything Model
• Authors: Yixin Zhang, Shen Zhao, Hanxue Gu, Maciej A. Mazurowski,
• Abstract summary: Deep neural networks (DNNs) have demonstrated exceptional performance across various image segmentation tasks.<n>To mitigate this challenge, alternative approaches such as using weak labels or less precise (noisy) annotations can be employed.<n>Noisy and weak labels are significantly quicker to generate, allowing for more annotated images within the same time frame.
• Score: 16.745318743249864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks (DNNs) have demonstrated exceptional performance across various image segmentation tasks. However, the process of preparing datasets for training segmentation DNNs is both labor-intensive and costly, as it typically requires pixel-level annotations for each object of interest. To mitigate this challenge, alternative approaches such as using weak labels (e.g., bounding boxes or scribbles) or less precise (noisy) annotations can be employed. Noisy and weak labels are significantly quicker to generate, allowing for more annotated images within the same time frame. However, the potential decrease in annotation quality may adversely impact the segmentation performance of the resulting model. In this study, we conducted a comprehensive cost-effectiveness evaluation on six variants of annotation strategies (9~10 sub-variants in total) across 4 datasets and conclude that the common practice of precisely outlining objects of interest is virtually never the optimal approach when annotation budget is limited. Both noisy and weak annotations showed usage cases that yield similar performance to the perfectly annotated counterpart, yet had significantly better cost-effectiveness. We hope our findings will help researchers be aware of the different available options and use their annotation budgets more efficiently, especially in cases where accurately acquiring labels for target objects is particularly costly. Our code will be made available on https://github.com/yzluka/AnnotationEfficiency2D.

Related papers

• Noisy Annotations in Semantic Segmentation [5.139071616097179]
  This study sheds light on the quality of segmentation masks produced by various models. It challenges the efficacy of popular methods designed to address learning with label noise.
  arXiv  Detail & Related papers  (2024-06-16T10:49:23Z)
• On-the-Fly Point Annotation for Fast Medical Video Labeling [1.890063512530524]
  In medical research, deep learning models rely on high-quality annotated data. The need to adjust two corners makes the process inherently frame-by-frame. We propose an on-the-fly method for live video annotation to enhance the annotation efficiency.
  arXiv  Detail & Related papers  (2024-04-22T16:59:43Z)
• Learning Tracking Representations from Single Point Annotations [49.47550029470299]
  We propose to learn tracking representations from single point annotations in a weakly supervised manner. Specifically, we propose a soft contrastive learning framework that incorporates target objectness prior to end-to-end contrastive learning.
  arXiv  Detail & Related papers  (2024-04-15T06:50:58Z)
• Deep Active Learning with Noisy Oracle in Object Detection [5.5165579223151795]
  We propose a composite active learning framework including a label review module for deep object detection. We show that utilizing part of the annotation budget to correct the noisy annotations partially in the active dataset leads to early improvements in model performance. In our experiments we achieve improvements of up to 4.5 mAP points of object detection performance by incorporating label reviews at equal annotation budget.
  arXiv  Detail & Related papers  (2023-09-30T13:28:35Z)
• TrueDeep: A systematic approach of crack detection with less data [0.0]
  We show that by incorporating domain knowledge along with deep learning architectures, we can achieve similar performance with less data. Our algorithms, developed with 23% of the overall data, have a similar performance on the test data and significantly better performance on multiple blind datasets.
  arXiv  Detail & Related papers  (2023-05-30T14:51:58Z)
• Knowledge Combination to Learn Rotated Detection Without Rotated Annotation [53.439096583978504]
  Rotated bounding boxes drastically reduce output ambiguity of elongated objects. Despite the effectiveness, rotated detectors are not widely employed. We propose a framework that allows the model to predict precise rotated boxes.
  arXiv  Detail & Related papers  (2023-04-05T03:07:36Z)
• Active Learning for Abstractive Text Summarization [50.79416783266641]
  We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
  arXiv  Detail & Related papers  (2023-01-09T10:33:14Z)
• Urban Scene Semantic Segmentation with Low-Cost Coarse Annotation [107.72926721837726]
  coarse annotation is a low-cost but highly effective alternative for training semantic segmentation models. We propose a coarse-to-fine self-training framework that generates pseudo labels for unlabeled regions of coarsely annotated data. Our method achieves a significantly better performance vs annotation cost tradeoff, yielding a comparable performance to fully annotated data with only a small fraction of the annotation budget.
  arXiv  Detail & Related papers  (2022-12-15T15:43:42Z)
• Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation [9.056814157662965]
  We show that modeling annotation biases, rather than ignoring them, poses a promising way of accounting for differences in annotation style across datasets. Next, we present an image-conditioning approach to model annotation styles that correlate with specific image features, potentially enabling detection biases to be more easily identified.
  arXiv  Detail & Related papers  (2022-10-31T15:28:49Z)
• Assisted Text Annotation Using Active Learning to Achieve High Quality with Little Effort [9.379650501033465]
  We propose a tool that enables researchers to create large, high-quality, annotated datasets with only a few manual annotations. We combine an active learning (AL) approach with a pre-trained language model to semi-automatically identify annotation categories. Our preliminary results show that employing AL strongly reduces the number of annotations for correct classification of even complex and subtle frames.
  arXiv  Detail & Related papers  (2021-12-15T13:14:58Z)
• A Positive/Unlabeled Approach for the Segmentation of Medical Sequences using Point-Wise Supervision [3.883460584034766]
  We propose a new method to efficiently segment medical imaging volumes or videos using point-wise annotations only. Our approach trains a deep learning model using an appropriate Positive/Unlabeled objective function using point-wise annotations. We show experimentally that our approach outperforms state-of-the-art methods tailored to the same problem.
  arXiv  Detail & Related papers  (2021-07-18T09:13:33Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• Temporal Calibrated Regularization for Robust Noisy Label Learning [60.90967240168525]
  Deep neural networks (DNNs) exhibit great success on many tasks with the help of large-scale well annotated datasets. However, labeling large-scale data can be very costly and error-prone so that it is difficult to guarantee the annotation quality. We propose a Temporal Calibrated Regularization (TCR) in which we utilize the original labels and the predictions in the previous epoch together.
  arXiv  Detail & Related papers  (2020-07-01T04:48:49Z)
• Improving Semantic Segmentation via Self-Training [75.07114899941095]
  We show that we can obtain state-of-the-art results using a semi-supervised approach, specifically a self-training paradigm. We first train a teacher model on labeled data, and then generate pseudo labels on a large set of unlabeled data. Our robust training framework can digest human-annotated and pseudo labels jointly and achieve top performances on Cityscapes, CamVid and KITTI datasets.
  arXiv  Detail & Related papers  (2020-04-30T17:09:17Z)
• Active Learning for Coreference Resolution using Discrete Annotation [76.36423696634584]
  We improve upon pairwise annotation for active learning in coreference resolution. We ask annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. In experiments with existing benchmark coreference datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour.
  arXiv  Detail & Related papers  (2020-04-28T17:17:11Z)
• Towards Using Count-level Weak Supervision for Crowd Counting [55.58468947486247]
  This paper studies the problem of weakly-supervised crowd counting which learns a model from only a small amount of location-level annotations (fully-supervised) but a large amount of count-level annotations (weakly-supervised) We devise a simple-yet-effective training strategy, namely Multiple Auxiliary Tasks Training (MATT), to construct regularizes for restricting the freedom of the generated density maps.
  arXiv  Detail & Related papers  (2020-02-29T02:58:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.