Beyond mAP: Towards better evaluation of instance segmentation

• URL: http://arxiv.org/abs/2207.01614v2
• Date: Mon, 20 Mar 2023 17:51:09 GMT
• Title: Beyond mAP: Towards better evaluation of instance segmentation
• Authors: Rohit Jena, Lukas Zhornyak, Nehal Doiphode, Pratik Chaudhari, Vivek Buch, James Gee, Jianbo Shi
• Abstract summary: Average Precision does not penalize duplicate predictions in the high-recall range. We propose two new measures to explicitly measure the amount of both spatial and categorical duplicate predictions. Our Semantic Sorting and NMS can be added as a plug-and-play module to mitigate hedged predictions and preserve AP.
• Score: 23.562251593257674
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Correctness of instance segmentation constitutes counting the number of objects, correctly localizing all predictions and classifying each localized prediction. Average Precision is the de-facto metric used to measure all these constituents of segmentation. However, this metric does not penalize duplicate predictions in the high-recall range, and cannot distinguish instances that are localized correctly but categorized incorrectly. This weakness has inadvertently led to network designs that achieve significant gains in AP but also introduce a large number of false positives. We therefore cannot rely on AP to choose a model that provides an optimal tradeoff between false positives and high recall. To resolve this dilemma, we review alternative metrics in the literature and propose two new measures to explicitly measure the amount of both spatial and categorical duplicate predictions. We also propose a Semantic Sorting and NMS module to remove these duplicates based on a pixel occupancy matching scheme. Experiments show that modern segmentation networks have significant gains in AP, but also contain a considerable amount of duplicates. Our Semantic Sorting and NMS can be added as a plug-and-play module to mitigate hedged predictions and preserve AP.

Related papers

• Semiparametric conformal prediction [79.6147286161434]
  Risk-sensitive applications require well-calibrated prediction sets over multiple, potentially correlated target variables. We treat the scores as random vectors and aim to construct the prediction set accounting for their joint correlation structure. We report desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• SortedAP: Rethinking evaluation metrics for instance segmentation [8.079566596963632]
  We show that most existing metrics have a limited resolution of segmentation quality. We propose a new metric called sortedAP, which strictly decreases with both object- and pixel-level imperfections.
  arXiv  Detail & Related papers  (2023-09-09T22:50:35Z)
• Ambiguity-Resistant Semi-Supervised Learning for Dense Object Detection [98.66771688028426]
  We propose a Ambiguity-Resistant Semi-supervised Learning (ARSL) for one-stage detectors. Joint-Confidence Estimation (JCE) is proposed to quantifies the classification and localization quality of pseudo labels. ARSL effectively mitigates the ambiguities and achieves state-of-the-art SSOD performance on MS COCO and PASCAL VOC.
  arXiv  Detail & Related papers  (2023-03-27T07:46:58Z)
• An Upper Bound for the Distribution Overlap Index and Its Applications [18.481370450591317]
  This paper proposes an easy-to-compute upper bound for the overlap index between two probability distributions. The proposed bound shows its value in one-class classification and domain shift analysis. Our work shows significant promise toward broadening the applications of overlap-based metrics.
  arXiv  Detail & Related papers  (2022-12-16T20:02:03Z)
• Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework [59.578339075658995]
  We propose a purely point-based framework for joint crowd counting and individual localization. We design an intuitive solution under this framework, which is called Point to Point Network (P2PNet)
  arXiv  Detail & Related papers  (2021-07-27T11:41:50Z)
• Uniformity in Heterogeneity:Diving Deep into Count Interval Partition for Crowd Counting [56.44300325295678]
  We propose a novel count interval partition criterion called Uniform Error Partition (UEP) MCP criterion selects the best count proxy for each interval to represent its count value during inference. We propose a simple yet effective model termed Uniform Error Partition Network (UEPNet)
  arXiv  Detail & Related papers  (2021-07-27T06:24:15Z)
• Which to Match? Selecting Consistent GT-Proposal Assignment for Pedestrian Detection [23.92066492219922]
  The fixed Intersection over Union (IoU) based assignment-regression manner still limits their performance. We introduce one geometric sensitive search algorithm as a new assignment and regression metric. Specifically, we boost the MR-FPPI under R$_75$ by 8.8% on Citypersons dataset.
  arXiv  Detail & Related papers  (2021-03-18T08:54:51Z)
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details [107.2722027807328]
  We find that the default implementation of AP is neither category independent, nor does it directly reward properly calibrated detectors. We show that the default implementation produces a gameable metric, where a simple, nonsensical re-ranking policy can improve AP by a large margin. We benchmark recent advances in large-vocabulary detection and find that many reported gains do not translate to improvements under our new per-class independent evaluation.
  arXiv  Detail & Related papers  (2021-02-01T18:56:02Z)
• Probabilistic Anchor Assignment with IoU Prediction for Object Detection [9.703212439661097]
  In object detection, determining which anchors to assign as positive or negative samples, known as anchor assignment, has been revealed as a core procedure that can significantly affect a model's performance. We propose a novel anchor assignment strategy that adaptively separates anchors into positive and negative samples for a ground truth bounding box according to the model's learning status.
  arXiv  Detail & Related papers  (2020-07-16T04:26:57Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.