When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images

• URL: http://arxiv.org/abs/2604.00651v1
• Date: Wed, 01 Apr 2026 09:01:43 GMT
• Title: When AI and Experts Agree on Error: Intrinsic Ambiguity in Dermatoscopic Images
• Abstract summary: We isolated a subset of images systematically misclassified across all CNN architectures.<n>Expert dermatologists evaluated these challenging cases alongside a control group.<n>Results revealed a collapse in human diagnostic performance on the AI-misclassified images.<n>We identified image quality as a primary driver of these dual systematic failures.
• Score: 37.92086450087161
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The integration of artificial intelligence (AI), particularly Convolutional Neural Networks (CNNs), into dermatological diagnosis demonstrates substantial clinical potential. While existing literature predominantly benchmarks algorithmic performance against human experts, our study adopts a novel perspective by investigating the intrinsic complexity of dermatoscopic images. Through rigorous experimentation with multiple CNN architectures, we isolated a subset of images systematically misclassified across all models-a phenomenon statistically proven to exceed random chance. To determine if these failures stem from algorithmic biases or inherent visual ambiguity, expert dermatologists independently evaluated these challenging cases alongside a control group. The results revealed a collapse in human diagnostic performance on the AI-misclassified images. First, agreement with ground-truth labels plummeted, with Cohen's kappa dropping to a mere 0.08 for the difficult images, compared to a 0.61 for the control group. Second, we observed a severe deterioration in expert consensus; inter-rater reliability among physicians fell from moderate concordance (Fleiss kappa = 0.456) on control images to only modest agreement (Fleiss kappa = 0.275) on difficult cases. We identified image quality as a primary driver of these dual systematic failures. To promote transparency and reproducibility, all data, code, and trained models have been made publicly available

Related papers

• Text-Guided Diffusion-Based Adversarial Attacks on Chest X-Ray Images [1.167393095135318]
  We propose a text-guided diffusion-based adversarial framework that optimize learnable text conditioning while keeping the diffusion generator and target classifier frozen.<n>Our approach consistently produced the greatest degradation in classifier performance, reducing AUROC to 0.3885-0.5646 in binary classification and 0.4441-0.4878 in the multi-disease setting.<n>These findings reveal a clinically important discrepancy between human and machine interpretation and demonstrate the need to extend medical AI robustness evaluation beyond conventional pixel-space attacks.
  arXiv  Detail & Related papers  (2026-08-29T22:19:04Z)
• BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases [59.19934491064188]
  Artificial intelligence (AI) has achieved remarkable success in medical imaging.<n>These models often perform inconsistently across real-world clinical settings.<n>We develop a benchmark of 85,355 CT scans that systematically evaluates 12 tumor-detection AI models.
  arXiv  Detail & Related papers  (2026-06-23T17:58:59Z)
• Anatomy-Aware Unsupervised Detection and Localization of Retinal Abnormalities in Optical Coherence Tomography [4.828654519487229]
  unsupervised anomaly detection framework learns the normative distribution of healthy retinal anatomy without lesion annotations.<n>We incorporate retinal layer-aware supervision and structured triplet learning to separate healthy from pathological representations.<n>On the Kermany dataset (AUROC: 0.799), our method substantially outperforms VAE, VQVAE, VQGAN, and f-AnoGAN baselines.
  arXiv  Detail & Related papers  (2026-04-24T01:09:05Z)
• GRASP-PsONet: Gradient-based Removal of Spurious Patterns for PsOriasis Severity Classification [0.0]
  We propose a framework to automatically flag problematic training images that introduce spurious correlations.<n>Removing 8.2% of flagged images improves model AUC-ROC by 5% (85% to 90%) on a held out test set.<n>When applied to a subset of training data rated by two dermatologists, the method identifies over 90% of cases with inter-rater disagreement.
  arXiv  Detail & Related papers  (2025-06-27T03:42:09Z)
• Lightweight Relational Embedding in Task-Interpolated Few-Shot Networks for Enhanced Gastrointestinal Disease Classification [0.0]
  Colon cancer detection is crucial for increasing patient survival rates.<n> colonoscopy is dependent on obtaining adequate and high-quality endoscopic images.<n>Few-Shot Learning architecture enables our model to rapidly adapt to unseen fine-grained endoscopic image patterns.<n>Our model demonstrated superior performance, achieving an accuracy of 90.1%, precision of 0.845, recall of 0.942, and an F1 score of 0.891.
  arXiv  Detail & Related papers  (2025-05-30T16:54:51Z)
• Epistemic Uncertainty for Generated Image Detection [107.62647907393377]
  We introduce a novel framework for AI-generated image detection through epistemic uncertainty, aiming to address critical security concerns in the era of generative models.<n>Our key insight stems from the observation that distributional discrepancies between training and testing data manifest distinctively in the epistemic uncertainty space of machine learning models.
  arXiv  Detail & Related papers  (2024-12-08T11:32:25Z)
• MedIAnomaly: A comparative study of anomaly detection in medical images [26.319602363581442]
  Anomaly detection (AD) aims at detecting abnormal samples that deviate from the expected normal patterns.<n>Despite the emergence of numerous methods for medical AD, the lack of a fair and comprehensive evaluation causes ambiguous conclusions.<n>This paper builds a benchmark with unified comparison to address this problem.
  arXiv  Detail & Related papers  (2024-04-06T06:18:11Z)
• Rescuing referral failures during automated diagnosis of domain-shifted medical images [17.349847762608086]
  We show that even state-of-the-art domain generalization approaches fail severely during referral when tested on medical images acquired from a different demographic or using a different technology. We evaluate novel combinations of robust generalization and post hoc referral approaches, that rescue these failures and achieve significant performance improvements.
  arXiv  Detail & Related papers  (2023-11-28T13:14:55Z)
• Uncertainty-inspired Open Set Learning for Retinal Anomaly Identification [71.06194656633447]
  We establish an uncertainty-inspired open-set (UIOS) model, which was trained with fundus images of 9 retinal conditions. Our UIOS model with thresholding strategy achieved an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set. UIOS correctly predicted high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images.
  arXiv  Detail & Related papers  (2023-04-08T10:47:41Z)
• GraVIS: Grouping Augmented Views from Independent Sources for Dermatology Analysis [52.04899592688968]
  We propose GraVIS, which is specifically optimized for learning self-supervised features from dermatology images. GraVIS significantly outperforms its transfer learning and self-supervised learning counterparts in both lesion segmentation and disease classification tasks.
  arXiv  Detail & Related papers  (2023-01-11T11:38:37Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [50.689585476660554]
  We propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• Anomaly Detection in Medical Imaging with Deep Perceptual Autoencoders [1.7277957019593995]
  We introduce a new powerful method of image anomaly detection. It relies on the classical autoencoder approach with a re-designed training pipeline. It outperforms state-of-the-art approaches in complex medical image analysis tasks.
  arXiv  Detail & Related papers  (2020-06-23T18:45:55Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.