A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis

• URL: http://arxiv.org/abs/2311.04157v3
• Date: Fri, 14 Jun 2024 17:28:14 GMT
• Title: A Simple Interpretable Transformer for Fine-Grained Image Classification and Analysis
• Abstract summary: We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR) We name our approach INterpretable TRansformer (INTR) which is fairly easy to implement and exhibits several compelling properties. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site.
• Score: 24.531871371861477
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present a novel usage of Transformers to make image classification interpretable. Unlike mainstream classifiers that wait until the last fully connected layer to incorporate class information to make predictions, we investigate a proactive approach, asking each class to search for itself in an image. We realize this idea via a Transformer encoder-decoder inspired by DEtection TRansformer (DETR). We learn "class-specific" queries (one for each class) as input to the decoder, enabling each class to localize its patterns in an image via cross-attention. We name our approach INterpretable TRansformer (INTR), which is fairly easy to implement and exhibits several compelling properties. We show that INTR intrinsically encourages each class to attend distinctively; the cross-attention weights thus provide a faithful interpretation of the prediction. Interestingly, via "multi-head" cross-attention, INTR could identify different "attributes" of a class, making it particularly suitable for fine-grained classification and analysis, which we demonstrate on eight datasets. Our code and pre-trained models are publicly accessible at the Imageomics Institute GitHub site: https://github.com/Imageomics/INTR.

Related papers

• Cross-Modal Prototype Alignment and Mixing for Training-Free Few-Shot Classification [52.48204114948899]
  We investigate the impact of directly mixing image and text prototypes for few-shot classification.<n>We show that mixing prototypes acts like a shrinkage estimator.<n>We propose projecting image prototypes onto the principal directions of the semantic text embedding space to obtain a text-aligned semantic image subspace.
  arXiv  Detail & Related papers  (2026-03-25T17:04:43Z)
• Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis [29.766516667434026]
  We present a novel approach to make pre-trained Vision Transformers (ViTs) interpretable for fine-grained analysis. Prompt-CAM learns class-specific prompts for a pre-trained ViT and uses the corresponding outputs for classification.
  arXiv  Detail & Related papers  (2025-01-16T07:07:41Z)
• Mining Open Semantics from CLIP: A Relation Transition Perspective for Few-Shot Learning [46.25534556546322]
  We propose to mine open semantics as anchors to perform a relation transition from image-anchor relationship to image-target relationship to make predictions. Our method performs favorably against previous state-of-the-arts considering few-shot classification settings.
  arXiv  Detail & Related papers  (2024-06-17T06:28:58Z)
• Text Descriptions are Compressive and Invariant Representations for Visual Learning [63.3464863723631]
  We show that an alternative approach, in line with humans' understanding of multiple visual features per class, can provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, textit SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors). This method first automatically generates multiple visual descriptions of each class via a large language model (LLM), then uses a VLM to translate these descriptions to a set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify
  arXiv  Detail & Related papers  (2023-07-10T03:06:45Z)
• Unicom: Universal and Compact Representation Learning for Image Retrieval [65.96296089560421]
  We cluster the large-scale LAION400M into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model. To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss. Our method significantly outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2023-04-12T14:25:52Z)
• Prediction Calibration for Generalized Few-shot Semantic Segmentation [101.69940565204816]
  Generalized Few-shot Semantic (GFSS) aims to segment each image pixel into either base classes with abundant training examples or novel classes with only a handful of (e.g., 1-5) training images per class. We build a cross-attention module that guides the classifier's final prediction using the fused multi-level features. Our PCN outperforms the state-the-art alternatives by large margins.
  arXiv  Detail & Related papers  (2022-10-15T13:30:12Z)
• iCAR: Bridging Image Classification and Image-text Alignment for Visual Recognition [33.2800417526215]
  Image classification, which classifies images by pre-defined categories, has been the dominant approach to visual representation learning over the last decade. Visual learning through image-text alignment, however, has emerged to show promising performance, especially for zero-shot recognition. We propose a deep fusion method with three adaptations that effectively bridge two learning tasks.
  arXiv  Detail & Related papers  (2022-04-22T15:27:21Z)
• Matching Feature Sets for Few-Shot Image Classification [22.84472344406448]
  We argue that a set-based representation intrinsically builds a richer representation of images from the base classes. Our approach, dubbed SetFeat, embeds shallow self-attention mechanisms inside existing encoder architectures.
  arXiv  Detail & Related papers  (2022-04-02T22:42:54Z)
• Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer [112.95747173442754]
  A few-shot semantic segmentation model is typically composed of a CNN encoder, a CNN decoder and a simple classifier. Most existing methods meta-learn all three model components for fast adaptation to a new class. In this work we propose to simplify the meta-learning task by focusing solely on the simplest component, the classifier.
  arXiv  Detail & Related papers  (2021-08-06T10:20:08Z)
• Segmenter: Transformer for Semantic Segmentation [79.9887988699159]
  We introduce Segmenter, a transformer model for semantic segmentation. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. It outperforms the state of the art on the challenging ADE20K dataset and performs on-par on Pascal Context and Cityscapes.
  arXiv  Detail & Related papers  (2021-05-12T13:01:44Z)
• Semantically Meaningful Class Prototype Learning for One-Shot Image Semantic Segmentation [58.96902899546075]
  One-shot semantic image segmentation aims to segment the object regions for the novel class with only one annotated image. Recent works adopt the episodic training strategy to mimic the expected situation at testing time. We propose to leverage the multi-class label information during the episodic training. It will encourage the network to generate more semantically meaningful features for each category.
  arXiv  Detail & Related papers  (2021-02-22T12:07:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.