Human-in-the-Loop Segmentation of Multi-species Coral Imagery

• URL: http://arxiv.org/abs/2404.09406v3
• Date: Tue, 12 Nov 2024 04:37:47 GMT
• Title: Human-in-the-Loop Segmentation of Multi-species Coral Imagery
• Authors: Scarlett Raine, Ross Marchant, Brano Kusy, Frederic Maire, Niko Suenderhauf, Tobias Fischer,
• Abstract summary: Point label propagation is a technique that uses existing images labeled with sparse points to create augmented ground truth data. We show that recent advances in large foundation models facilitate the creation of augmented ground truth masks. We present a labeling method based on human-in-the-loop principles, which greatly enhances annotation efficiency.
• Score: 3.3564744382205127
• License:
• Abstract: Marine surveys by robotic underwater and surface vehicles result in substantial quantities of coral reef imagery, however labeling these images is expensive and time-consuming for domain experts. Point label propagation is a technique that uses existing images labeled with sparse points to create augmented ground truth data, which can be used to train a semantic segmentation model. In this work, we show that recent advances in large foundation models facilitate the creation of augmented ground truth masks using only features extracted by the denoised version of the DINOv2 foundation model and K-Nearest Neighbors (KNN), without any pre-training. For images with extremely sparse labels, we present a labeling method based on human-in-the-loop principles, which greatly enhances annotation efficiency: in the case that there are 5 point labels per image, our human-in-the-loop method outperforms the prior state-of-the-art by 14.2% for pixel accuracy and 19.7% for mIoU; and by 8.9% and 18.3% if there are 10 point labels. When human-in-the-loop labeling is not available, using the denoised DINOv2 features with a KNN still improves on the prior state-of-the-art by 2.7% for pixel accuracy and 5.8% for mIoU (5 grid points). On the semantic segmentation task, we outperform the prior state-of-the-art by 8.8% for pixel accuracy and by 13.5% for mIoU when only 5 point labels are used for point label propagation. Additionally, we perform a comprehensive study into the impacts of the point label placement style and the number of points on the point label propagation quality, and make several recommendations for improving the efficiency of labeling images with points.

Related papers

• Adaptive Anchor Label Propagation for Transductive Few-Shot Learning [18.29463308334406]
  Few-shot learning addresses the issue of classifying images using limited labeled data. We propose a novel algorithm that adapts the feature embeddings of the labeled data by minimizing a differentiable loss function. Our algorithm outperforms the standard label propagation algorithm by as much as 7% and 2% in the 1-shot and 5-shot settings respectively.
  arXiv  Detail & Related papers  (2023-10-30T20:29:31Z)
• Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation [58.03255076119459]
  We address the task of weakly-supervised few-shot image classification and segmentation, by leveraging a Vision Transformer (ViT) Our proposed method takes token representations from the self-supervised ViT and leverages their correlations, via self-attention, to produce classification and segmentation predictions. Experiments on Pascal-5i and COCO-20i demonstrate significant performance gains in a variety of supervision settings.
  arXiv  Detail & Related papers  (2023-07-07T06:16:43Z)
• CorrMatch: Label Propagation via Correlation Matching for Semi-Supervised Semantic Segmentation [73.89509052503222]
  This paper presents a simple but performant semi-supervised semantic segmentation approach, called CorrMatch. We observe that the correlation maps not only enable clustering pixels of the same category easily but also contain good shape information. We propose to conduct pixel propagation by modeling the pairwise similarities of pixels to spread the high-confidence pixels and dig out more. Then, we perform region propagation to enhance the pseudo labels with accurate class-agnostic masks extracted from the correlation maps.
  arXiv  Detail & Related papers  (2023-06-07T10:02:29Z)
• SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentation [52.62441404064957]
  Domain adaptive semantic segmentation attempts to make satisfactory dense predictions on an unlabeled target domain by utilizing the model trained on a labeled source domain. Many methods tend to alleviate noisy pseudo labels, however, they ignore intrinsic connections among cross-domain pixels with similar semantic concepts. We propose Semantic-Guided Pixel Contrast (SePiCo), a novel one-stage adaptation framework that highlights the semantic concepts of individual pixel.
  arXiv  Detail & Related papers  (2022-04-19T11:16:29Z)
• Point Label Aware Superpixels for Multi-species Segmentation of Underwater Imagery [4.195806160139487]
  Monitoring coral reefs using underwater vehicles increases the range of marine surveys and availability of historical ecological data. We propose a point label aware method for propagating labels within superpixel regions to obtain augmented ground truth for training a semantic segmentation model. Our method outperforms prior methods on the UCSD Mosaics dataset by 3.62% for pixel accuracy and 8.35% for mean IoU for the label propagation task.
  arXiv  Detail & Related papers  (2022-02-27T23:46:43Z)
• LabOR: Labeling Only if Required for Domain Adaptive Semantic Segmentation [79.96052264984469]
  We propose a human-in-the-loop approach to adaptively give scarce labels to points that a UDA model is uncertain about. We show the advantages of this new framework for domain adaptive semantic segmentation while minimizing human labor costs.
  arXiv  Detail & Related papers  (2021-08-12T07:35:40Z)
• Pseudo Pixel-level Labeling for Images with Evolving Content [5.573543601558405]
  We propose a pseudo-pixel-level label generation technique to reduce the amount of effort for manual annotation of images. We train two semantic segmentation models with VGG and ResNet backbones on images labeled using our pseudo labeling method and those of a state-of-the-art method. The results indicate that using our pseudo-labels instead of those generated using the state-of-the-art method in the training process improves the mean-IoU and the frequency-weighted-IoU of the VGG and ResNet-based semantic segmentation models by 3.36%, 2.58%, 10
  arXiv  Detail & Related papers  (2021-05-20T18:14:19Z)
• Towards Good Practices for Efficiently Annotating Large-Scale Image Classification Datasets [90.61266099147053]
  We investigate efficient annotation strategies for collecting multi-class classification labels for a large collection of images. We propose modifications and best practices aimed at minimizing human labeling effort. Simulated experiments on a 125k image subset of the ImageNet100 show that it can be annotated to 80% top-1 accuracy with 0.35 annotations per image on average.
  arXiv  Detail & Related papers  (2021-04-26T16:29:32Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision [12.284208932393073]
  This paper presents a novel procedure for producing semantic segmentation from images given some point level annotations. We propose training a CNN that is normally fully supervised using our pseudo labels in place of ground truth labels. Our method achieves state of the art results for point supervised semantic segmentation on the PASCAL VOC 2012 dataset citeeveringham2010pascal, even outperforming state of the art methods for stronger bounding box and squiggle supervision.
  arXiv  Detail & Related papers  (2020-07-10T21:25:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.