Comparison of Evaluation Metrics for Landmark Detection in CMR Images

• URL: http://arxiv.org/abs/2201.10410v1
• Date: Tue, 25 Jan 2022 15:58:30 GMT
• Title: Comparison of Evaluation Metrics for Landmark Detection in CMR Images
• Authors: Sven Koehler, Lalith Sharan, Julian Kuhm, Arman Ghanaat, Jelizaveta Gordejeva, Nike K. Simon, Niko M. Grell, Florian Andr\'e, Sandy Engelhardt
• Abstract summary: We extend the public ACDC dataset with additional labels of the right ventricular insertion points. We compare different variants of a heatmap-based landmark detection pipeline. Preliminary results indicate that a combination of different metrics is necessary.
• Score: 0.8219153654616499
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Cardiac Magnetic Resonance (CMR) images are widely used for cardiac diagnosis and ventricular assessment. Extracting specific landmarks like the right ventricular insertion points is of importance for spatial alignment and 3D modeling. The automatic detection of such landmarks has been tackled by multiple groups using Deep Learning, but relatively little attention has been paid to the failure cases of evaluation metrics in this field. In this work, we extended the public ACDC dataset with additional labels of the right ventricular insertion points and compare different variants of a heatmap-based landmark detection pipeline. In this comparison, we demonstrate very likely pitfalls of apparently simple detection and localisation metrics which highlights the importance of a clear detection strategy and the definition of an upper limit for localisation-based metrics. Our preliminary results indicate that a combination of different metrics is necessary, as they yield different winners for method comparison. Additionally, they highlight the need of a comprehensive metric description and evaluation standardisation, especially for the error cases where no metrics could be computed or where no lower/upper boundary of a metric exists. Code and labels: https://github.com/Cardio-AI/rvip_landmark_detection

Related papers

• Every Component Counts: Rethinking the Measure of Success for Medical Semantic Segmentation in Multi-Instance Segmentation Tasks [60.80828925396154]
  We present Connected-Component(CC)-Metrics, a novel semantic segmentation evaluation protocol. We motivate this setup in the common medical scenario of semantic segmentation in a full-body PET/CT. We show how existing semantic segmentation metrics suffer from a bias towards larger connected components.
  arXiv  Detail & Related papers  (2024-10-24T12:26:05Z)
• EchoGLAD: Hierarchical Graph Neural Networks for Left Ventricle Landmark Detection on Echocardiograms [21.537833605289816]
  We introduce an echocardiogram-based, hierarchical graph neural network (GNN) for left ventricle landmark detection (EchoGLAD) Our main contributions are 1) a hierarchical graph representation learning framework for multi-resolution landmark detection via GNNs; 2) induced hierarchical supervision at different levels of granularity using a multi-level loss.
  arXiv  Detail & Related papers  (2023-07-23T05:31:47Z)
• Range-Based Equal Error Rate for Spoof Localization [43.75986914767975]
  Spoof localization is a crucial task that aims to locate spoofs in partially spoofed audio. The equal error rate (EER) is widely used to measure performance for such biometric scenarios. We upgrade point-based EER to range-based EER and compare it with the classical point-based EER.
  arXiv  Detail & Related papers  (2023-05-28T14:46:54Z)
• Robust Detection Outcome: A Metric for Pathology Detection in Medical Images [6.667150890634173]
  Robust Detection Outcome (RoDeO) is a novel metric for evaluating algorithms for pathology detection in medical images. RoDeO evaluates different errors directly and individually, and reflects clinical needs better than current metrics.
  arXiv  Detail & Related papers  (2023-03-03T13:45:13Z)
• Self-Supervised Equivariant Regularization Reconciles Multiple Instance Learning: Joint Referable Diabetic Retinopathy Classification and Lesion Segmentation [3.1671604920729224]
  Lesion appearance is a crucial clue for medical providers to distinguish referable diabetic retinopathy (rDR) from non-referable DR. Most existing large-scale DR datasets contain only image-level labels rather than pixel-based annotations. This paper leverages self-supervised equivariant learning and attention-based multi-instance learning to tackle this problem. We conduct extensive validation experiments on the Eyepacs dataset, achieving an area under the receiver operating characteristic curve (AU ROC) of 0.958, outperforming current state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2022-10-12T06:26:05Z)
• The impact of using voxel-level segmentation metrics on evaluating multifocal prostate cancer localisation [8.035409264165937]
  Dice similarity coefficient (DSC) and Hausdorff distance (HD) are widely used for evaluating medical image segmentation. This work first proposes a new asymmetric detection metric, adapting those used in object detection, for planning prostate cancer procedures. We report pairwise agreement and correlation 1) between DSC and HD, and 2) between voxel-level DSC and recall-controlled precision at lesion-level.
  arXiv  Detail & Related papers  (2022-03-30T15:57:20Z)
• Supervision by Registration and Triangulation for Landmark Detection [70.13440728689231]
  We present Supervision by Registration and Triangulation (SRT), an unsupervised approach that utilizes unlabeled multi-view video to improve the accuracy and precision of landmark detectors. Being able to utilize unlabeled data enables our detectors to learn from massive amounts of unlabeled data freely available.
  arXiv  Detail & Related papers  (2021-01-25T02:48:21Z)
• Dense Label Encoding for Boundary Discontinuity Free Rotation Detection [69.75559390700887]
  This paper explores a relatively less-studied methodology based on classification. We propose new techniques to push its frontier in two aspects. Experiments and visual analysis on large-scale public datasets for aerial images show the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-11-19T05:42:02Z)
• Making Affine Correspondences Work in Camera Geometry Computation [62.7633180470428]
  Local features provide region-to-region rather than point-to-point correspondences. We propose guidelines for effective use of region-to-region matches in the course of a full model estimation pipeline. Experiments show that affine solvers can achieve accuracy comparable to point-based solvers at faster run-times.
  arXiv  Detail & Related papers  (2020-07-20T12:07:48Z)
• Rethinking Localization Map: Towards Accurate Object Perception with Self-Enhancement Maps [78.2581910688094]
  This work introduces a novel self-enhancement method to harvest accurate object localization maps and object boundaries with only category labels as supervision. In particular, the proposed Self-Enhancement Maps achieve the state-of-the-art localization accuracy of 54.88% on ILSVRC.
  arXiv  Detail & Related papers  (2020-06-09T12:35:55Z)
• Do Public Datasets Assure Unbiased Comparisons for Registration Evaluation? [96.53940048041248]
  We use the variogram to screen the manually annotated landmarks in two datasets used to benchmark registration in image-guided neurosurgeries. Using variograms, we identified potentially problematic cases and had them examined by experienced radiologists.
  arXiv  Detail & Related papers  (2020-03-20T20:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.