Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models

• URL: http://arxiv.org/abs/2310.03182v1
• Date: Wed, 4 Oct 2023 21:57:09 GMT
• Title: Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models
• Authors: An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amilcare Gentili, Chun-Nan Hsu, Jingbo Shang, Julian McAuley
• Abstract summary: We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
• Score: 49.95603725998561
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Medical image classification is a critical problem for healthcare, with the potential to alleviate the workload of doctors and facilitate diagnoses of patients. However, two challenges arise when deploying deep learning models to real-world healthcare applications. First, neural models tend to learn spurious correlations instead of desired features, which could fall short when generalizing to new domains (e.g., patients with different ages). Second, these black-box models lack interpretability. When making diagnostic predictions, it is important to understand why a model makes a decision for trustworthy and safety considerations. In this paper, to address these two limitations, we propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model. We systematically evaluate our method on eight medical image classification datasets to verify its effectiveness. On challenging datasets with strong confounding factors, our method can mitigate spurious correlations thus substantially outperform standard visual encoders and other baselines. Finally, we show how classification with a small number of concepts brings a level of interpretability for understanding model decisions through case studies in real medical data.

Related papers

• A Two-Step Concept-Based Approach for Enhanced Interpretability and Trust in Skin Lesion Diagnosis [6.6635650150737815]
  Concept Bottleneck Models (CBMs) offer inherent interpretability by constraining the final disease prediction on a set of human-understandable concepts. We introduce a novel two-step methodology that addresses both of these challenges. By simulating the two stages of a CBM, we utilize a pretrained Vision Language Model (VLM) to automatically predict clinical concepts, and a Large Language Model (LLM) to generate disease diagnoses.
  arXiv  Detail & Related papers  (2024-11-08T14:52:42Z)
• DiffExplainer: Unveiling Black Box Models Via Counterfactual Generation [11.201840101870808]
  We propose an agent model capable of generating counterfactual images that prompt different decisions when plugged into a black box model. By employing this agent model, we can uncover influential image patterns that impact the black model's final predictions. We validated our approach in the rigorous domain of medical prognosis tasks.
  arXiv  Detail & Related papers  (2024-06-21T14:27:02Z)
• A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis [48.84443450990355]
  Deep networks have achieved broad success in analyzing natural images, when applied to medical scans, they often fail in unexcepted situations. We investigate this challenge and focus on model sensitivity to domain shifts, such as data sampled from different hospitals or data confounded by demographic variables such as sex, race, etc, in the context of chest X-rays and skin lesion images. Taking inspiration from medical training, we propose giving deep networks a prior grounded in explicit medical knowledge communicated in natural language.
  arXiv  Detail & Related papers  (2024-05-23T17:55:02Z)
• MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment [4.861768967055006]
  We propose a multi-modal explainable disease diagnosis framework that meticulously aligns medical images and clinical-related concepts semantically at multiple strata. Our method, while preserving model interpretability, attains high performance and label efficiency for concept detection and disease diagnosis.
  arXiv  Detail & Related papers  (2024-01-16T17:45:01Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Pixel-Level Explanation of Multiple Instance Learning Models in Biomedical Single Cell Images [52.527733226555206]
  We investigate the use of four attribution methods to explain a multiple instance learning models. We study two datasets of acute myeloid leukemia with over 100 000 single cell images. We compare attribution maps with the annotations of a medical expert to see how the model's decision-making differs from the human standard.
  arXiv  Detail & Related papers  (2023-03-15T14:00:11Z)
• Towards Trustable Skin Cancer Diagnosis via Rewriting Model's Decision [12.306688233127312]
  We introduce a human-in-the-loop framework in the model training process. Our method can automatically discover confounding factors. It is capable of learning confounding concepts using easily obtained concept exemplars.
  arXiv  Detail & Related papers  (2023-03-02T01:02:18Z)
• Review of Disentanglement Approaches for Medical Applications -- Towards Solving the Gordian Knot of Generative Models in Healthcare [3.5586630313792513]
  We give a comprehensive overview of popular generative models, like Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs) and Flow-based Models. After introducing the theoretical frameworks, we give an overview of recent medical applications and discuss the impact and importance of disentanglement approaches for medical applications.
  arXiv  Detail & Related papers  (2022-03-21T17:06:22Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)
• Semi-supervised Medical Image Classification with Relation-driven Self-ensembling Model [71.80319052891817]
  We present a relation-driven semi-supervised framework for medical image classification. It exploits the unlabeled data by encouraging the prediction consistency of given input under perturbations. Our method outperforms many state-of-the-art semi-supervised learning methods on both single-label and multi-label image classification scenarios.
  arXiv  Detail & Related papers  (2020-05-15T06:57:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.