Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient

• URL: http://arxiv.org/abs/2103.06205v4
• Date: Tue, 2 May 2023 13:42:03 GMT
• Title: Are we using appropriate segmentation metrics? Identifying correlates of human expert perception for CNN training beyond rolling the DICE coefficient
• Authors: Florian Kofler, Ivan Ezhov, Fabian Isensee, Fabian Balsiger, Christoph Berger, Maximilian Koerner, Beatrice Demiray, Julia Rackerseder, Johannes Paetzold, Hongwei Li, Suprosanna Shit, Richard McKinley, Marie Piraud, Spyridon Bakas, Claus Zimmer, Nassir Navab, Jan Kirschke, Benedikt Wiestler, Bjoern Menze
• Abstract summary: We conduct psychophysical experiments for two complex biomedical semantic segmentation problems. We discover that current standard metrics and loss functions correlate only moderately with the segmentation quality assessment of experts. It is often unclear how to optimize abstract metrics, such as human expert perception, in convolutional neural network (CNN) training.
• Score: 30.31460995779947
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metrics optimized in complex machine learning tasks are often selected in an ad-hoc manner. It is unknown how they align with human expert perception. We explore the correlations between established quantitative segmentation quality metrics and qualitative evaluations by professionally trained human raters. Therefore, we conduct psychophysical experiments for two complex biomedical semantic segmentation problems. We discover that current standard metrics and loss functions correlate only moderately with the segmentation quality assessment of experts. Importantly, this effect is particularly pronounced for clinically relevant structures, such as the enhancing tumor compartment of glioma in brain magnetic resonance and grey matter in ultrasound imaging. It is often unclear how to optimize abstract metrics, such as human expert perception, in convolutional neural network (CNN) training. To cope with this challenge, we propose a novel strategy employing techniques of classical statistics to create complementary compound loss functions to better approximate human expert perception. Across all rating experiments, human experts consistently scored computer-generated segmentations better than the human-curated reference labels. Our results, therefore, strongly question many current practices in medical image segmentation and provide meaningful cues for future research.

Related papers

• Multi-task Explainable Skin Lesion Classification [54.76511683427566]
  We propose a few-shot-based approach for skin lesions that generalizes well with few labelled data. The proposed approach comprises a fusion of a segmentation network that acts as an attention module and classification network.
  arXiv  Detail & Related papers  (2023-10-11T05:49:47Z)
• Discrimination of Radiologists Utilizing Eye-Tracking Technology and Machine Learning: A Case Study [0.9142067094647588]
  This study presents a novel discretized feature encoding based on binning fixation data for efficient geometric alignment. The encoded features of the eye-fixation data are employed by machine learning classifiers to discriminate between faculty and trainee radiologists.
  arXiv  Detail & Related papers  (2023-08-04T23:51:47Z)
• K-Space-Aware Cross-Modality Score for Synthesized Neuroimage Quality Assessment [71.27193056354741]
  The problem of how to assess cross-modality medical image synthesis has been largely unexplored. We propose a new metric K-CROSS to spur progress on this challenging problem. K-CROSS uses a pre-trained multi-modality segmentation network to predict the lesion location.
  arXiv  Detail & Related papers  (2023-07-10T01:26:48Z)
• Problem-Dependent Power of Quantum Neural Networks on Multi-Class Classification [83.20479832949069]
  Quantum neural networks (QNNs) have become an important tool for understanding the physical world, but their advantages and limitations are not fully understood. Here we investigate the problem-dependent power of QCs on multi-class classification tasks. Our work sheds light on the problem-dependent power of QNNs and offers a practical tool for evaluating their potential merit.
  arXiv  Detail & Related papers  (2022-12-29T10:46:40Z)
• Pain level and pain-related behaviour classification using GRU-based sparsely-connected RNNs [61.080598804629375]
  People with chronic pain unconsciously adapt specific body movements to protect themselves from injury or additional pain. Because there is no dedicated benchmark database to analyse this correlation, we considered one of the specific circumstances that potentially influence a person's biometrics during daily activities. We proposed a sparsely-connected recurrent neural networks (s-RNNs) ensemble with the gated recurrent unit (GRU) that incorporates multiple autoencoders. We conducted several experiments which indicate that the proposed method outperforms the state-of-the-art approaches in classifying both pain level and pain-related behaviour.
  arXiv  Detail & Related papers  (2022-12-20T12:56:28Z)
• Interaction of a priori Anatomic Knowledge with Self-Supervised Contrastive Learning in Cardiac Magnetic Resonance Imaging [0.7387261884863349]
  Self-supervised contrastive learning has been shown to boost performance in several medical imaging tasks. In this work, we evaluate the optimal method of incorporating prior knowledge of anatomy into a SSCL training paradigm. We find that using a priori knowledge of anatomy can greatly improve the downstream diagnostic performance.
  arXiv  Detail & Related papers  (2022-05-25T01:33:37Z)
• Medulloblastoma Tumor Classification using Deep Transfer Learning with Multi-Scale EfficientNets [63.62764375279861]
  We propose an end-to-end MB tumor classification and explore transfer learning with various input sizes and matching network dimensions. Using a data set with 161 cases, we demonstrate that pre-trained EfficientNets with larger input resolutions lead to significant performance improvements.
  arXiv  Detail & Related papers  (2021-09-10T13:07:11Z)
• Deep Learning Based Cardiac MRI Segmentation: Do We Need Experts? [12.36854197042851]
  We show that a segmentation neural network trained on non-expert groundtruth data is, to all practical purposes, as good as on expert groundtruth data. We highlight an opportunity for the efficient and cheap creation of annotations for cardiac datasets.
  arXiv  Detail & Related papers  (2021-07-23T20:10:58Z)
• A Global Benchmark of Algorithms for Segmenting Late Gadolinium-Enhanced Cardiac Magnetic Resonance Imaging [90.29017019187282]
  " 2018 Left Atrium Challenge" using 154 3D LGE-MRIs, currently the world's largest cardiac LGE-MRI dataset. Analyse of the submitted algorithms using technical and biological metrics was performed. Results show the top method achieved a dice score of 93.2% and a mean surface to a surface distance of 0.7 mm.
  arXiv  Detail & Related papers  (2020-04-26T08:49:17Z)
• Spinal Metastases Segmentation in MR Imaging using Deep Convolutional Neural Networks [0.0]
  This study's objective was to segment spinal metastases in diagnostic MR images using a deep learning-based approach. We used a U-Net like architecture trained with 40 clinical cases including both, lytic and sclerotic lesion types and various MR sequences. Compared to expertly annotated lesion segmentations, the experiments yielded promising results with average Dice scores up to 77.6% and mean sensitivity rates up to 78.9%.
  arXiv  Detail & Related papers  (2020-01-08T10:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.