Systematic Evaluation of Attribution Methods: Eliminating Threshold Bias and Revealing Method-Dependent Performance Patterns

• URL: http://arxiv.org/abs/2509.03176v1
• Date: Wed, 03 Sep 2025 09:50:31 GMT
• Title: Systematic Evaluation of Attribution Methods: Eliminating Threshold Bias and Revealing Method-Dependent Performance Patterns
• Abstract summary: Current protocols binarize attribution maps at single thresholds, where threshold choice alone can alter rankings by over 200 percentage points.<n>We address this flaw with a threshold-free framework that computes Area Under the Curve for Intersection over Union (AUC-IoU)<n> XRAI achieves 31% improvement over LIME and 204% over vanilla Integrated Gradients, with size-stratified analysis revealing performance variations up to 269% across lesion scales.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attribution methods explain neural network predictions by identifying influential input features, but their evaluation suffers from threshold selection bias that can reverse method rankings and undermine conclusions. Current protocols binarize attribution maps at single thresholds, where threshold choice alone can alter rankings by over 200 percentage points. We address this flaw with a threshold-free framework that computes Area Under the Curve for Intersection over Union (AUC-IoU), capturing attribution quality across the full threshold spectrum. Evaluating seven attribution methods on dermatological imaging, we show single-threshold metrics yield contradictory results, while threshold-free evaluation provides reliable differentiation. XRAI achieves 31% improvement over LIME and 204% over vanilla Integrated Gradients, with size-stratified analysis revealing performance variations up to 269% across lesion scales. These findings establish methodological standards that eliminate evaluation artifacts and enable evidence-based method selection. The threshold-free framework provides both theoretical insight into attribution behavior and practical guidance for robust comparison in medical imaging and beyond.

Related papers

• Automated binary classification of hazelnut X-ray images: A deep-learning benchmark for quality assessment [33.72751145910978]
  Non-destructive X-ray imaging can reveal internal hazelnut defects that are difficult to detect by external inspection alone.<n>We present a benchmark for binary hazelnut quality classification based on 799 segmented single- Kernel X-ray images.
  arXiv  Detail & Related papers  (2026-08-12T07:55:22Z)
• Deep EM with Hierarchical Latent Label Modelling for Multi-Site Prostate Lesion Segmentation [12.493998903996589]
  We propose a hierarchical expectation-maximisation (HierEM) framework that alternates between inferring a voxel-wise posterior distribution over the latent mask, and training a CNN using this posterior as a soft target.<n> Experiments on three cohorts demonstrate that the proposed hierarchical EM framework enhances cross-site generalisation.
  arXiv  Detail & Related papers  (2026-03-15T15:03:20Z)
• Channel Selected Stratified Nested Cross Validation for Clinically Relevant EEG Based Parkinsons Disease Detection [2.384534878752428]
  We propose a unified evaluation framework grounded in nested cross validation and incorporating three complementary safeguards.<n>A convolutional neural network trained under this framework achieved 80.6% accuracy and demonstrated state of the art performance under held out population block testing.
  arXiv  Detail & Related papers  (2025-12-28T23:34:38Z)
• Flexible Nonparametric Inference for Causal Effects under the Front-Door Model [2.6900047294457683]
  We develop novel one-step and targeted minimum loss-based estimators for both the average treatment effect and the average treatment effect on the treated under front-door assumptions.<n>Our estimators are built on multiple parameterizations of the observed data distribution, including approaches that avoid mediator density entirely.<n>We show how these constraints can be leveraged to improve the efficiency of causal effect estimators.
  arXiv  Detail & Related papers  (2023-12-15T22:04:53Z)
• Post-hoc Orthogonalization for Mitigation of Protected Feature Bias in CXR Embeddings [10.209740962369453]
  We analyze and remove protected feature effects in chest radiograph embeddings of deep learning models. Experiments reveal a significant influence of protected features on predictions of pathologies.
  arXiv  Detail & Related papers  (2023-11-02T15:59:00Z)
• B-Learner: Quasi-Oracle Bounds on Heterogeneous Causal Effects Under Hidden Confounding [51.74479522965712]
  We propose a meta-learner called the B-Learner, which can efficiently learn sharp bounds on the CATE function under limits on hidden confounding. We prove its estimates are valid, sharp, efficient, and have a quasi-oracle property with respect to the constituent estimators under more general conditions than existing methods.
  arXiv  Detail & Related papers  (2023-04-20T18:07:19Z)
• Better Understanding Differences in Attribution Methods via Systematic Evaluations [57.35035463793008]
  Post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. We propose three novel evaluation schemes to more reliably measure the faithfulness of those methods. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models.
  arXiv  Detail & Related papers  (2023-03-21T14:24:58Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• Causal Inference under Data Restrictions [0.0]
  This dissertation focuses on modern causal inference under uncertainty and data restrictions. It includes applications to neoadjuvant clinical trials, distributed data networks, and robust individualized decision making.
  arXiv  Detail & Related papers  (2023-01-20T20:14:32Z)
• Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise [62.997667081978825]
  In high-risk environments, deep learning models need to be able to judge their uncertainty and reject inputs when there is a significant chance of misclassification. We conduct a rigorous evaluation of the most commonly used uncertainty and robustness methods for the classification of Whole Slide Images. We observe that ensembles of methods generally lead to better uncertainty estimates as well as an increased robustness towards domain shifts and label noise.
  arXiv  Detail & Related papers  (2023-01-03T11:34:36Z)
• Fake It Till You Make It: Near-Distribution Novelty Detection by Score-Based Generative Models [54.182955830194445]
  existing models either fail or face a dramatic drop under the so-called near-distribution" setting. We propose to exploit a score-based generative model to produce synthetic near-distribution anomalous data. Our method improves the near-distribution novelty detection by 6% and passes the state-of-the-art by 1% to 5% across nine novelty detection benchmarks.
  arXiv  Detail & Related papers  (2022-05-28T02:02:53Z)
• Assessment of Treatment Effect Estimators for Heavy-Tailed Data [70.72363097550483]
  A central obstacle in the objective assessment of treatment effect (TE) estimators in randomized control trials (RCTs) is the lack of ground truth (or validation set) to test their performance. We provide a novel cross-validation-like methodology to address this challenge. We evaluate our methodology across 709 RCTs implemented in the Amazon supply chain.
  arXiv  Detail & Related papers  (2021-12-14T17:53:01Z)
• Exploring Instance-Level Uncertainty for Medical Detection [16.637462795585773]
  In this work, we explore to augment a 2.5D detection CNN with two different bounding-box-level (or instance-level) uncertainty estimates. Experiments are conducted for lung nodule detection on LUNA16 dataset, a task where significant semantic ambiguities can exist. Results show that our method improves the evaluating score from 84.57% to 88.86% by utilizing a combination of both types of variances.
  arXiv  Detail & Related papers  (2020-12-23T18:46:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.