Understanding metric-related pitfalls in image analysis validation
- URL: http://arxiv.org/abs/2302.01790v4
- Date: Fri, 23 Feb 2024 13:37:33 GMT
- Title: Understanding metric-related pitfalls in image analysis validation
- Authors: Annika Reinke, Minu D. Tizabi, Michael Baumgartner, Matthias
Eisenmann, Doreen Heckmann-N\"otzel, A. Emre Kavur, Tim R\"adsch, Carole H.
Sudre, Laura Acion, Michela Antonelli, Tal Arbel, Spyridon Bakas, Arriel
Benis, Matthew Blaschko, Florian Buettner, M. Jorge Cardoso, Veronika
Cheplygina, Jianxu Chen, Evangelia Christodoulou, Beth A. Cimini, Gary S.
Collins, Keyvan Farahani, Luciana Ferrer, Adrian Galdran, Bram van Ginneken,
Ben Glocker, Patrick Godau, Robert Haase, Daniel A. Hashimoto, Michael M.
Hoffman, Merel Huisman, Fabian Isensee, Pierre Jannin, Charles E. Kahn,
Dagmar Kainmueller, Bernhard Kainz, Alexandros Karargyris, Alan
Karthikesalingam, Hannes Kenngott, Jens Kleesiek, Florian Kofler, Thijs Kooi,
Annette Kopp-Schneider, Michal Kozubek, Anna Kreshuk, Tahsin Kurc, Bennett A.
Landman, Geert Litjens, Amin Madani, Klaus Maier-Hein, Anne L. Martel, Peter
Mattson, Erik Meijering, Bjoern Menze, Karel G.M. Moons, Henning M\"uller,
Brennan Nichyporuk, Felix Nickel, Jens Petersen, Susanne M. Rafelski, Nasir
Rajpoot, Mauricio Reyes, Michael A. Riegler, Nicola Rieke, Julio
Saez-Rodriguez, Clara I. S\'anchez, Shravya Shetty, Maarten van Smeden,
Ronald M. Summers, Abdel A. Taha, Aleksei Tiulpin, Sotirios A. Tsaftaris, Ben
Van Calster, Ga\"el Varoquaux, Manuel Wiesenfarth, Ziv R. Yaniv, Paul F.
J\"ager, Lena Maier-Hein
- Abstract summary: This work provides the first comprehensive common point of access to information on pitfalls related to validation metrics in image analysis.
Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy.
- Score: 59.15220116166561
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Validation metrics are key for the reliable tracking of scientific progress
and for bridging the current chasm between artificial intelligence (AI)
research and its translation into practice. However, increasing evidence shows
that particularly in image analysis, metrics are often chosen inadequately in
relation to the underlying research problem. This could be attributed to a lack
of accessibility of metric-related knowledge: While taking into account the
individual strengths, weaknesses, and limitations of validation metrics is a
critical prerequisite to making educated choices, the relevant knowledge is
currently scattered and poorly accessible to individual researchers. Based on a
multi-stage Delphi process conducted by a multidisciplinary expert consortium
as well as extensive community feedback, the present work provides the first
reliable and comprehensive common point of access to information on pitfalls
related to validation metrics in image analysis. Focusing on biomedical image
analysis but with the potential of transfer to other fields, the addressed
pitfalls generalize across application domains and are categorized according to
a newly created, domain-agnostic taxonomy. To facilitate comprehension,
illustrations and specific examples accompany each pitfall. As a structured
body of information accessible to researchers of all levels of expertise, this
work enhances global comprehension of a key topic in image analysis validation.
Related papers
- Unity in Diversity: Multi-expert Knowledge Confrontation and Collaboration for Generalizable Vehicle Re-identification [32.80872775195836]
Generalizable vehicle re-identification (ReID) aims to enable the well-trained model in diverse source domains to broadly adapt to unknown target domains.
It still faces the challenges of domain shift problem and has difficulty accurately generalizing to unknown target domains.
This paper proposes the two-stage Multi-expert Knowledge Confrontation and Collaboration (MiKeCoCo) method.
arXiv Detail & Related papers (2024-07-10T04:06:39Z) - Human Body Pose Estimation for Gait Identification: A Comprehensive
Survey of Datasets and Models [4.17510581764131]
Person identification is a problem that has received substantial attention, particularly in security domains.
There are several review studies addressing person identification such as the utilization of facial images, silhouette images, and wearable sensor.
Despite skeleton-based person identification gaining popularity while overcoming the challenges of traditional approaches, existing survey studies lack the comprehensive review of skeleton-based approaches to gait identification.
arXiv Detail & Related papers (2023-05-23T07:30:00Z) - Modeling Multiple Views via Implicitly Preserving Global Consistency and
Local Complementarity [61.05259660910437]
We propose a global consistency and complementarity network (CoCoNet) to learn representations from multiple views.
On the global stage, we reckon that the crucial knowledge is implicitly shared among views, and enhancing the encoder to capture such knowledge can improve the discriminability of the learned representations.
Lastly on the local stage, we propose a complementarity-factor, which joints cross-view discriminative knowledge, and it guides the encoders to learn not only view-wise discriminability but also cross-view complementary information.
arXiv Detail & Related papers (2022-09-16T09:24:00Z) - Variational Distillation for Multi-View Learning [104.17551354374821]
We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning.
Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
arXiv Detail & Related papers (2022-06-20T03:09:46Z) - Metrics reloaded: Recommendations for image analysis validation [59.60445111432934]
Metrics Reloaded is a comprehensive framework guiding researchers in the problem-aware selection of metrics.
The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint.
Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics.
arXiv Detail & Related papers (2022-06-03T15:56:51Z) - Link Analysis meets Ontologies: Are Embeddings the Answer? [0.0]
We present a systematic evaluation of whether structure-only link analysis methods can offer a scalable means to detecting possible anomalies.
We demonstrate that structure-only link analysis could offer scalable anomaly detection for a subset of the data sets.
This is one of the most extensive systematic studies of the applicability of different types of link analysis methods across semantic resources from different domains.
arXiv Detail & Related papers (2021-11-23T08:05:43Z) - Common Limitations of Image Processing Metrics: A Picture Story [58.83274952067888]
This document focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task.
The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.
arXiv Detail & Related papers (2021-04-12T17:03:42Z) - Evaluation of Complexity Measures for Deep Learning Generalization in
Medical Image Analysis [77.34726150561087]
PAC-Bayes flatness-based and path norm-based measures produce the most consistent explanation for the combination of models and data.
We also investigate the use of multi-task classification and segmentation approach for breast images.
arXiv Detail & Related papers (2021-03-04T20:58:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.