Explainable AI Methods for Neuroimaging: Systematic Failures of Common Tools, the Need for Domain-Specific Validation, and a Proposal for Safe Application

• URL: http://arxiv.org/abs/2508.02560v1
• Date: Mon, 04 Aug 2025 16:14:15 GMT
• Title: Explainable AI Methods for Neuroimaging: Systematic Failures of Common Tools, the Need for Domain-Specific Validation, and a Proposal for Safe Application
• Authors: Nys Tjade Siegel, James H. Cole, Mohamad Habes, Stefan Haufe, Kerstin Ritter, Marc-André Schulz,
• Abstract summary: We performed the first large-scale, systematic comparison of Explainable AI (XAI) methods on 45,000 structural brain MRIs.<n>Our analysis reveals systematic failures in two of the most widely used methods.<n>These failures stem from a domain mismatch, where methods with design principles tailored to natural images require substantial adaptation for neuroimaging data.
• Score: 1.0000511213628438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Trustworthy interpretation of deep learning models is critical for neuroimaging applications, yet commonly used Explainable AI (XAI) methods lack rigorous validation, risking misinterpretation. We performed the first large-scale, systematic comparison of XAI methods on ~45,000 structural brain MRIs using a novel XAI validation framework. This framework establishes verifiable ground truth by constructing prediction tasks with known signal sources - from localized anatomical features to subject-specific clinical lesions - without artificially altering input images. Our analysis reveals systematic failures in two of the most widely used methods: GradCAM consistently failed to localize predictive features, while Layer-wise Relevance Propagation generated extensive, artifactual explanations that suggest incompatibility with neuroimaging data characteristics. Our results indicate that these failures stem from a domain mismatch, where methods with design principles tailored to natural images require substantial adaptation for neuroimaging data. In contrast, the simpler, gradient-based method SmoothGrad, which makes fewer assumptions about data structure, proved consistently accurate, suggesting its conceptual simplicity makes it more robust to this domain shift. These findings highlight the need for domain-specific adaptation and validation of XAI methods, suggest that interpretations from prior neuroimaging studies using standard XAI methodology warrant re-evaluation, and provide urgent guidance for practical application of XAI in neuroimaging.

Related papers

• Certified Neural Approximations of Nonlinear Dynamics [52.79163248326912]
  In safety-critical contexts, the use of neural approximations requires formal bounds on their closeness to the underlying system.<n>We propose a novel, adaptive, and parallelizable verification method based on certified first-order models.
  arXiv  Detail & Related papers  (2025-05-21T13:22:20Z)
• Trustworthy image-to-image translation: evaluating uncertainty calibration in unpaired training scenarios [0.0]
  Mammographic screening is an effective method for detecting breast cancer, facilitating early diagnosis.<n>Deep neural networks have been shown effective in some studies, but their tendency to overfit leaves considerable risk for poor generalisation and misdiagnosis.<n>Data augmentation schemes based on unpaired neural style transfer models have been proposed that improve generalisability.<n>We evaluate their performance when trained on image patches parsed from three open access mammography datasets and one non-medical image dataset.
  arXiv  Detail & Related papers  (2025-01-29T11:09:50Z)
• SEANN: A Domain-Informed Neural Network for Epidemiological Insights [0.9749638953163389]
  We introduce SEANN, a novel approach for informed agnostics that leverages a prevalent form of domain-specific knowledge: Pooled Effect Sizes (PES)<n>PESs are commonly found in published Meta-Analysis studies, in different forms, and represent a quantitative form of a scientific consensus.<n>We experimentally demonstrate significant improvements in the generalizability of predictive performances and the scientific plausibility of extracted relationships.
  arXiv  Detail & Related papers  (2025-01-17T16:01:05Z)
• Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection [58.87142367781417]
  A naively trained detector tends to favor overfitting to the limited and monotonous fake patterns, causing the feature space to become highly constrained and low-ranked.<n>One potential remedy is incorporating the pre-trained knowledge within the vision foundation models to expand the feature space.<n>By freezing the principal components and adapting only the remained components, we preserve the pre-trained knowledge while learning fake patterns.
  arXiv  Detail & Related papers  (2024-11-23T19:10:32Z)
• IBO: Inpainting-Based Occlusion to Enhance Explainable Artificial Intelligence Evaluation in Histopathology [1.9440228513607511]
  Inpainting-Based Occlusion (IBO) is a novel strategy that utilizes a Denoising Diffusion Probabilistic Model to inpaint occluded regions. We evaluate IBO through two phases: first, by assessing perceptual similarity using the Learned Perceptual Image Patch Similarity (LPIPS) metric, and second, by quantifying the impact on model predictions through Area Under the Curve (AUC) analysis.
  arXiv  Detail & Related papers  (2024-08-29T09:57:55Z)
• DeepTechnome: Mitigating Unknown Bias in Deep Learning Based Assessment of CT Images [44.62475518267084]
  We debias deep learning models during training against unknown bias. We use control regions as surrogates that carry information regarding the bias. Applying the proposed method to learn from data exhibiting a strong bias, it near-perfectly recovers the classification performance observed when training with corresponding unbiased data.
  arXiv  Detail & Related papers  (2022-05-26T12:18:48Z)
• Ultrasound Signal Processing: From Models to Deep Learning [64.56774869055826]
  Medical ultrasound imaging relies heavily on high-quality signal processing to provide reliable and interpretable image reconstructions. Deep learning based methods, which are optimized in a data-driven fashion, have gained popularity. A relatively new paradigm combines the power of the two: leveraging data-driven deep learning, as well as exploiting domain knowledge.
  arXiv  Detail & Related papers  (2022-04-09T13:04:36Z)
• Facial Anatomical Landmark Detection using Regularized Transfer Learning with Application to Fetal Alcohol Syndrome Recognition [24.27777060287004]
  Fetal alcohol syndrome (FAS) caused by prenatal alcohol exposure can result in a series of cranio-facial anomalies. Anatomical landmark detection is important to detect the presence of FAS associated facial anomalies. Current deep learning-based heatmap regression methods designed for facial landmark detection in natural images assume availability of large datasets. We develop a new regularized transfer learning approach that exploits the knowledge of a network learned on large facial recognition datasets.
  arXiv  Detail & Related papers  (2021-09-12T11:05:06Z)
• Unsupervised anomaly detection in digital pathology using GANs [4.318555434063274]
  We propose a new unsupervised learning approach for anomaly detection in histopathology data based on generative adversarial networks (GANs) Compared to the existing GAN-based methods that have been used in medical imaging, the proposed approach improves significantly on performance for pathology data.
  arXiv  Detail & Related papers  (2021-03-16T10:10:12Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Fader Networks for domain adaptation on fMRI: ABIDE-II study [68.5481471934606]
  We use 3D convolutional autoencoders to build the domain irrelevant latent space image representation and demonstrate this method to outperform existing approaches on ABIDE data.
  arXiv  Detail & Related papers  (2020-10-14T16:50:50Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.