Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability

• URL: http://arxiv.org/abs/2503.20301v1
• Date: Wed, 26 Mar 2025 07:59:04 GMT
• Title: Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability
• Authors: Jianyang Zhang, Qianli Luo, Guowu Yang, Wenjing Yang, Weide Liu, Guosheng Lin, Fengmao Lv,
• Abstract summary: We propose the Attribute-formed Language Bottleneck Model (ALBM) to achieve interpretable image recognition.<n>ALBM organizes concepts in the attribute-formed class-specific space, where concepts are descriptions of specific attributes for specific classes.<n>To further improve interpretability, we propose Visual Attribute Prompt Learning (VAPL) to extract visual features on fine-grained attributes.
• Score: 54.420663939897686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language Bottleneck Models (LBMs) are proposed to achieve interpretable image recognition by classifying images based on textual concept bottlenecks. However, current LBMs simply list all concepts together as the bottleneck layer, leading to the spurious cue inference problem and cannot generalized to unseen classes. To address these limitations, we propose the Attribute-formed Language Bottleneck Model (ALBM). ALBM organizes concepts in the attribute-formed class-specific space, where concepts are descriptions of specific attributes for specific classes. In this way, ALBM can avoid the spurious cue inference problem by classifying solely based on the essential concepts of each class. In addition, the cross-class unified attribute set also ensures that the concept spaces of different classes have strong correlations, as a result, the learned concept classifier can be easily generalized to unseen classes. Moreover, to further improve interpretability, we propose Visual Attribute Prompt Learning (VAPL) to extract visual features on fine-grained attributes. Furthermore, to avoid labor-intensive concept annotation, we propose the Description, Summary, and Supplement (DSS) strategy to automatically generate high-quality concept sets with a complete and precise attribute. Extensive experiments on 9 widely used few-shot benchmarks demonstrate the interpretability, transferability, and performance of our approach. The code and collected concept sets are available at https://github.com/tiggers23/ALBM.

Related papers

• Compositional Caching for Training-free Open-vocabulary Attribute Detection [65.46250297408974]
  We present Compositional Caching (ComCa), a training-free method for open-vocabulary attribute detection.<n>ComCa requires only the list of target attributes and objects as input, using them to populate an auxiliary cache of images.<n>Experiments on public datasets demonstrate that ComCa significantly outperforms zero-shot and cache-based baselines.
  arXiv  Detail & Related papers  (2025-03-24T21:00:37Z)
• Real Classification by Description: Extending CLIP's Limits of Part Attributes Recognition [1.2499537119440243]
  We tackle zero shot "real" classification by description, a novel task that evaluates the ability of Vision-Language Models (VLMs) to classify objects based solely on descriptive attributes, excluding object class names.<n>We release description data for six popular fine-grained benchmarks, which omit object names to encourage genuine zero-shot learning.<n>We introduce a modified CLIP architecture that leverages multiple resolutions to improve the detection of fine-grained part attributes.
  arXiv  Detail & Related papers  (2024-12-18T15:28:08Z)
• Multi-modal Attribute Prompting for Vision-Language Models [40.39559705414497]
  Pre-trained Vision-Language Models (VLMs) exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. We propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment.
  arXiv  Detail & Related papers  (2024-03-01T01:28:10Z)
• Hierarchical Visual Primitive Experts for Compositional Zero-Shot Learning [52.506434446439776]
  Compositional zero-shot learning (CZSL) aims to recognize compositions with prior knowledge of known primitives (attribute and object) We propose a simple and scalable framework called Composition Transformer (CoT) to address these issues. Our method achieves SoTA performance on several benchmarks, including MIT-States, C-GQA, and VAW-CZSL.
  arXiv  Detail & Related papers  (2023-08-08T03:24:21Z)
• Text Descriptions are Compressive and Invariant Representations for Visual Learning [63.3464863723631]
  We show that an alternative approach, in line with humans' understanding of multiple visual features per class, can provide compelling performance in the robust few-shot learning setting. In particular, we introduce a novel method, textit SLR-AVD (Sparse Logistic Regression using Augmented Visual Descriptors). This method first automatically generates multiple visual descriptions of each class via a large language model (LLM), then uses a VLM to translate these descriptions to a set of visual feature embeddings of each image, and finally uses sparse logistic regression to select a relevant subset of these features to classify
  arXiv  Detail & Related papers  (2023-07-10T03:06:45Z)
• Translational Concept Embedding for Generalized Compositional Zero-shot Learning [73.60639796305415]
  Generalized compositional zero-shot learning means to learn composed concepts of attribute-object pairs in a zero-shot fashion. This paper introduces a new approach, termed translational concept embedding, to solve these two difficulties in a unified framework.
  arXiv  Detail & Related papers  (2021-12-20T21:27:51Z)
• Rich Semantics Improve Few-shot Learning [49.11659525563236]
  We show that by using 'class-level' language descriptions, that can be acquired with minimal annotation cost, we can improve the few-shot learning performance. We develop a Transformer based forward and backward encoding mechanism to relate visual and semantic tokens.
  arXiv  Detail & Related papers  (2021-04-26T16:48:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.