One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images

• URL: http://arxiv.org/abs/2107.05047v1
• Date: Sun, 11 Jul 2021 13:43:02 GMT
• Title: One Map Does Not Fit All: Evaluating Saliency Map Explanation on Multi-Modal Medical Images
• Authors: Weina Jin, Xiaoxiao Li, Ghassan Hamarneh
• Abstract summary: For medical images, saliency maps are the most common form of explanation. Our evaluations show that although most saliency map methods captured modality importance information in general, most of them failed to highlight modality-specific important features consistently and precisely.
• Score: 22.672569495620895
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Being able to explain the prediction to clinical end-users is a necessity to leverage the power of AI models for clinical decision support. For medical images, saliency maps are the most common form of explanation. The maps highlight important features for AI model's prediction. Although many saliency map methods have been proposed, it is unknown how well they perform on explaining decisions on multi-modal medical images, where each modality/channel carries distinct clinical meanings of the same underlying biomedical phenomenon. Understanding such modality-dependent features is essential for clinical users' interpretation of AI decisions. To tackle this clinically important but technically ignored problem, we propose the MSFI (Modality-Specific Feature Importance) metric to examine whether saliency maps can highlight modality-specific important features. MSFI encodes the clinical requirements on modality prioritization and modality-specific feature localization. Our evaluations on 16 commonly used saliency map methods, including a clinician user study, show that although most saliency map methods captured modality importance information in general, most of them failed to highlight modality-specific important features consistently and precisely. The evaluation results guide the choices of saliency map methods and provide insights to propose new ones targeting clinical applications.

Related papers

• Few Exemplar-Based General Medical Image Segmentation via Domain-Aware Selective Adaptation [28.186785488818135]
  Medical image segmentation poses challenges due to domain gaps, data modality variations, and dependency on domain knowledge or experts. We introduce a domain-aware selective adaptation approach to adapt the general knowledge learned from a large model trained with natural images to the corresponding medical domains/modalities.
  arXiv  Detail & Related papers  (2024-10-11T21:00:57Z)
• DSCENet: Dynamic Screening and Clinical-Enhanced Multimodal Fusion for MPNs Subtype Classification [8.95697172316912]
  We propose a Dynamic Screening and Clinical-Enhanced Network (DSCENet) for the subtype classification of MPNs on the multimodal fusion of whole slide images (WSIs) and clinical information. Our approach has been validated on the real clinical data, achieving an increase of 7.91% AUC and 16.89% accuracy compared with the previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-11T04:13:58Z)
• Key Patches Are All You Need: A Multiple Instance Learning Framework For Robust Medical Diagnosis [15.964609888720315]
  We propose to limit the amount of information deep learning models use to reach the final classification, by using a multiple instance learning framework. We evaluate our framework on two medical applications: skin cancer diagnosis using dermoscopy and breast cancer diagnosis using mammography.
  arXiv  Detail & Related papers  (2024-05-02T18:21:25Z)
• Pixel-Level Explanation of Multiple Instance Learning Models in Biomedical Single Cell Images [52.527733226555206]
  We investigate the use of four attribution methods to explain a multiple instance learning models. We study two datasets of acute myeloid leukemia with over 100 000 single cell images. We compare attribution maps with the annotations of a medical expert to see how the model's decision-making differs from the human standard.
  arXiv  Detail & Related papers  (2023-03-15T14:00:11Z)
• Morphology-Aware Interactive Keypoint Estimation [32.52024944963992]
  Diagnosis based on medical images often involves manual annotation of anatomical keypoints. We propose a novel deep neural network that automatically detects and refines the anatomical keypoints through a user-interactive system.
  arXiv  Detail & Related papers  (2022-09-15T09:27:14Z)
• Evaluating Explainable AI on a Multi-Modal Medical Imaging Task: Can Existing Algorithms Fulfill Clinical Requirements? [42.75635888823057]
  Heatmap is a form of explanation that highlights important features for AI models' prediction. It is unknown how well heatmaps perform on explaining decisions on multi-modal medical images. We propose the modality-specific feature importance (MSFI) metric to tackle this clinically important but technically ignored problem.
  arXiv  Detail & Related papers  (2022-03-12T17:18:16Z)
• Semi-Supervised Variational Reasoning for Medical Dialogue Generation [70.838542865384]
  Two key characteristics are relevant for medical dialogue generation: patient states and physician actions. We propose an end-to-end variational reasoning approach to medical dialogue generation. A physician policy network composed of an action-classifier and two reasoning detectors is proposed for augmented reasoning ability.
  arXiv  Detail & Related papers  (2021-05-13T04:14:35Z)
• Inheritance-guided Hierarchical Assignment for Clinical Automatic Diagnosis [50.15205065710629]
  Clinical diagnosis, which aims to assign diagnosis codes for a patient based on the clinical note, plays an essential role in clinical decision-making. We propose a novel framework to combine the inheritance-guided hierarchical assignment and co-occurrence graph propagation for clinical automatic diagnosis.
  arXiv  Detail & Related papers  (2021-01-27T13:16:51Z)
• Explaining Clinical Decision Support Systems in Medical Imaging using Cycle-Consistent Activation Maximization [112.2628296775395]
  Clinical decision support using deep neural networks has become a topic of steadily growing interest. clinicians are often hesitant to adopt the technology because its underlying decision-making process is considered to be intransparent and difficult to comprehend. We propose a novel decision explanation scheme based on CycleGAN activation which generates high-quality visualizations of classifier decisions even in smaller data sets.
  arXiv  Detail & Related papers  (2020-10-09T14:39:27Z)
• Weakly supervised multiple instance learning histopathological tumor segmentation [51.085268272912415]
  We propose a weakly supervised framework for whole slide imaging segmentation. We exploit a multiple instance learning scheme for training models. The proposed framework has been evaluated on multi-locations and multi-centric public data from The Cancer Genome Atlas and the PatchCamelyon dataset.
  arXiv  Detail & Related papers  (2020-04-10T13:12:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.