Interpretable Image Classification with Adaptive Prototype-based Vision Transformers

• URL: http://arxiv.org/abs/2410.20722v1
• Date: Mon, 28 Oct 2024 04:33:28 GMT
• Title: Interpretable Image Classification with Adaptive Prototype-based Vision Transformers
• Authors: Chiyu Ma, Jon Donnelly, Wenjun Liu, Soroush Vosoughi, Cynthia Rudin, Chaofan Chen,
• Abstract summary: We present ProtoViT, a method for interpretable image classification combining deep learning and case-based reasoning. Our model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes. Our experiments show that our model can generally achieve higher performance than the existing prototype based models.
• Score: 37.62530032165594
• License:
• Abstract: We present ProtoViT, a method for interpretable image classification combining deep learning and case-based reasoning. This method classifies an image by comparing it to a set of learned prototypes, providing explanations of the form ``this looks like that.'' In our model, a prototype consists of \textit{parts}, which can deform over irregular geometries to create a better comparison between images. Unlike existing models that rely on Convolutional Neural Network (CNN) backbones and spatially rigid prototypes, our model integrates Vision Transformer (ViT) backbones into prototype based models, while offering spatially deformed prototypes that not only accommodate geometric variations of objects but also provide coherent and clear prototypical feature representations with an adaptive number of prototypical parts. Our experiments show that our model can generally achieve higher performance than the existing prototype based models. Our comprehensive analyses ensure that the prototypes are consistent and the interpretations are faithful.

Related papers

• Mind the Gap Between Prototypes and Images in Cross-domain Finetuning [64.97317635355124]
  We propose a contrastive prototype-image adaptation (CoPA) to adapt different transformations respectively for prototypes and images. Experiments on Meta-Dataset demonstrate that CoPA achieves the state-of-the-art performance more efficiently.
  arXiv  Detail & Related papers  (2024-10-16T11:42:11Z)
• Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation [7.372346036256517]
  Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. We propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. Experiments conducted on Pascal VOC, Cityscapes, and ADE20K demonstrate that the proposed method increases model sparsity, improves interpretability over existing prototype-based methods, and narrows the performance gap with the non-interpretable counterpart models.
  arXiv  Detail & Related papers  (2024-09-14T17:52:59Z)
• Query-guided Prototype Evolution Network for Few-Shot Segmentation [85.75516116674771]
  We present a new method that integrates query features into the generation process of foreground and background prototypes. Experimental results on the PASCAL-$5i$ and mirroring-$20i$ datasets attest to the substantial enhancements achieved by QPENet.
  arXiv  Detail & Related papers  (2024-03-11T07:50:40Z)
• ProtoP-OD: Explainable Object Detection with Prototypical Parts [0.0]
  This paper introduces an extension to detection transformers that constructs prototypical local features and uses them in object detection. The proposed extension consists of a bottleneck module, the prototype neck, that computes a discretized representation of prototype activations.
  arXiv  Detail & Related papers  (2024-02-29T13:25:15Z)
• This Looks Like Those: Illuminating Prototypical Concepts Using Multiple Visualizations [19.724372592639774]
  ProtoConcepts is a method for interpretable image classification combining deep learning and case-based reasoning. Our proposed method modifies the architecture of prototype-based networks to instead learn concepts which are visualized using multiple image patches. Our experiments show that our this looks like those'' reasoning process can be applied as a modification to a wide range of existing prototypical image classification networks.
  arXiv  Detail & Related papers  (2023-10-28T04:54:48Z)
• Rethinking Semantic Segmentation: A Prototype View [126.59244185849838]
  We present a nonparametric semantic segmentation model based on non-learnable prototypes. Our framework yields compelling results over several datasets. We expect this work will provoke a rethink of the current de facto semantic segmentation model design.
  arXiv  Detail & Related papers  (2022-03-28T21:15:32Z)
• Interpretable Image Classification with Differentiable Prototypes Assignment [7.660883761395447]
  We introduce ProtoPool, an interpretable image classification model with a pool of prototypes shared by the classes. It is obtained by introducing a fully differentiable assignment of prototypes to particular classes. We show that ProtoPool obtains state-of-the-art accuracy on the CUB-200-2011 and the Stanford Cars datasets, substantially reducing the number of prototypes.
  arXiv  Detail & Related papers  (2021-12-06T10:03:32Z)
• Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes [7.8515366468594765]
  We present a deformable part network (Deformable ProtoPNet) that integrates the power of deep learning and the interpretability of case-based reasoning. This model classifies input images by comparing them with prototypes learned during training, yielding explanations in the form of "this looks like that"
  arXiv  Detail & Related papers  (2021-11-29T22:38:13Z)
• Meta Internal Learning [88.68276505511922]
  Internal learning for single-image generation is a framework, where a generator is trained to produce novel images based on a single image. We propose a meta-learning approach that enables training over a collection of images, in order to model the internal statistics of the sample image more effectively. Our results show that the models obtained are as suitable as single-image GANs for many common image applications.
  arXiv  Detail & Related papers  (2021-10-06T16:27:38Z)
• Learning Sparse Prototypes for Text Generation [120.38555855991562]
  Prototype-driven text generation is inefficient at test time as a result of needing to store and index the entire training corpus. We propose a novel generative model that automatically learns a sparse prototype support set that achieves strong language modeling performance. In experiments, our model outperforms previous prototype-driven language models while achieving up to a 1000x memory reduction.
  arXiv  Detail & Related papers  (2020-06-29T19:41:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.