Common Limitations of Image Processing Metrics: A Picture Story
- URL: http://arxiv.org/abs/2104.05642v8
- Date: Wed, 6 Dec 2023 16:00:22 GMT
- Title: Common Limitations of Image Processing Metrics: A Picture Story
- Authors: Annika Reinke, Minu D. Tizabi, Carole H. Sudre, Matthias Eisenmann,
Tim R\"adsch, Michael Baumgartner, Laura Acion, Michela Antonelli, Tal Arbel,
Spyridon Bakas, Peter Bankhead, Arriel Benis, Matthew Blaschko, Florian
Buettner, M. Jorge Cardoso, Jianxu Chen, Veronika Cheplygina, Evangelia
Christodoulou, Beth Cimini, Gary S. Collins, Sandy Engelhardt, Keyvan
Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken, Ben Glocker,
Patrick Godau, Robert Haase, Fred Hamprecht, Daniel A. Hashimoto, Doreen
Heckmann-N\"otzel, Peter Hirsch, Michael M. Hoffman, Merel Huisman, Fabian
Isensee, Pierre Jannin, Charles E. Kahn, Dagmar Kainmueller, Bernhard Kainz,
Alexandros Karargyris, Alan Karthikesalingam, A. Emre Kavur, Hannes Kenngott,
Jens Kleesiek, Andreas Kleppe, Sven Kohler, Florian Kofler, Annette
Kopp-Schneider, Thijs Kooi, Michal Kozubek, Anna Kreshuk, Tahsin Kurc,
Bennett A. Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L.
Martel, Peter Mattson, Erik Meijering, Bjoern Menze, David Moher, Karel G.M.
Moons, Henning M\"uller, Brennan Nichyporuk, Felix Nickel, M. Alican Noyan,
Jens Petersen, Gorkem Polat, Susanne M. Rafelski, Nasir Rajpoot, Mauricio
Reyes, Nicola Rieke, Michael Riegler, Hassan Rivaz, Julio Saez-Rodriguez,
Clara I. S\'anchez, Julien Schroeter, Anindo Saha, M. Alper Selver, Lalith
Sharan, Shravya Shetty, Maarten van Smeden, Bram Stieltjes, Ronald M.
Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben Van
Calster, Ga\"el Varoquaux, Manuel Wiesenfarth, Ziv R. Yaniv, Paul J\"ager,
Lena Maier-Hein
- Abstract summary: This document focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task.
The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.
- Score: 58.83274952067888
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: While the importance of automatic image analysis is continuously increasing,
recent meta-research revealed major flaws with respect to algorithm validation.
Performance metrics are particularly key for meaningful, objective, and
transparent performance assessment and validation of the used automatic
algorithms, but relatively little attention has been given to the practical
pitfalls when using specific metrics for a given image analysis task. These are
typically related to (1) the disregard of inherent metric properties, such as
the behaviour in the presence of class imbalance or small target structures,
(2) the disregard of inherent data set properties, such as the non-independence
of the test cases, and (3) the disregard of the actual biomedical domain
interest that the metrics should reflect. This living dynamically document has
the purpose to illustrate important limitations of performance metrics commonly
applied in the field of image analysis. In this context, it focuses on
biomedical image analysis problems that can be phrased as image-level
classification, semantic segmentation, instance segmentation, or object
detection task. The current version is based on a Delphi process on metrics
conducted by an international consortium of image analysis experts from more
than 60 institutions worldwide.
Related papers
- Visual Context-Aware Person Fall Detection [52.49277799455569]
We present a segmentation pipeline to semi-automatically separate individuals and objects in images.
Background objects such as beds, chairs, or wheelchairs can challenge fall detection systems, leading to false positive alarms.
We demonstrate that object-specific contextual transformations during training effectively mitigate this challenge.
arXiv Detail & Related papers (2024-04-11T19:06:36Z) - Unsupervised Domain Adaptation for Medical Image Segmentation via
Feature-space Density Matching [0.0]
This paper presents an unsupervised domain adaptation approach for semantic segmentation.
We match the target data distribution to the source in the feature space, particularly when the number of target samples is limited.
We demonstrate the efficacy of our proposed approach on 2 datasets, multisite prostate MRI and histopathology images.
arXiv Detail & Related papers (2023-05-09T22:24:46Z) - Understanding metric-related pitfalls in image analysis validation [59.15220116166561]
This work provides the first comprehensive common point of access to information on pitfalls related to validation metrics in image analysis.
Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy.
arXiv Detail & Related papers (2023-02-03T14:57:40Z) - Metrics reloaded: Recommendations for image analysis validation [59.60445111432934]
Metrics Reloaded is a comprehensive framework guiding researchers in the problem-aware selection of metrics.
The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint.
Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics.
arXiv Detail & Related papers (2022-06-03T15:56:51Z) - How can we learn (more) from challenges? A statistical approach to
driving future algorithm development [1.0690055408831725]
We present a statistical framework for learning from challenges and instantiate it for the specific task of instrument instance segmentation in laparoscopic videos.
Based on 51,542 meta data performed on 2,728 images, we applied our approach to the results of the Robust Medical Instrument Challenge (ROBUST-MIS) challenge 2019.
Our method development, tailored to the specific remaining issues, yielded a deep learning model with state-of-the-art overall performance and specific strengths in the processing of images in which previous methods tended to fail.
arXiv Detail & Related papers (2021-06-17T08:12:37Z) - Self-Path: Self-supervision for Classification of Pathology Images with
Limited Annotations [4.713391888104896]
We propose a self-supervised CNN approach to leverage unlabeled data for learning generalizable and domain invariant representations in pathology images.
We introduce novel domain specific self-supervision tasks that leverage contextual, multi-resolution and semantic features in pathology images.
arXiv Detail & Related papers (2020-08-12T21:02:32Z) - Phase Consistent Ecological Domain Adaptation [76.75730500201536]
We focus on the task of semantic segmentation, where annotated synthetic data are aplenty, but annotating real data is laborious.
The first criterion, inspired by visual psychophysics, is that the map between the two image domains be phase-preserving.
The second criterion aims to leverage ecological statistics, or regularities in the scene which are manifest in any image of it, regardless of the characteristics of the illuminant or the imaging sensor.
arXiv Detail & Related papers (2020-04-10T06:58:03Z) - Panoptic Feature Fusion Net: A Novel Instance Segmentation Paradigm for
Biomedical and Biological Images [91.41909587856104]
We present a Panoptic Feature Fusion Net (PFFNet) that unifies the semantic and instance features in this work.
Our proposed PFFNet contains a residual attention feature fusion mechanism to incorporate the instance prediction with the semantic features.
It outperforms several state-of-the-art methods on various biomedical and biological datasets.
arXiv Detail & Related papers (2020-02-15T09:19:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.