Multimodal Concept Bottleneck Models

• URL: http://arxiv.org/abs/2606.19882v1
• Date: Thu, 18 Jun 2026 07:41:50 GMT
• Title: Multimodal Concept Bottleneck Models
• Abstract summary: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.<n>We propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP.<n>Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks.
• Score: 19.41992921538502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts. However, existing CBMs are constrained in their ability to generalize beyond a fixed set of predefined classes and the risk of non-concept information leakage, where predictive signals outside the intended concepts are inadvertently exploited. In this paper, we propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP. MM-CBM utilizes dual Concept Bottleneck Layers (CBLs) to align both the image and text embeddings into interpretable features. This allows us to perform new vision tasks like zero-shot classification or image retrieval in an interpretable way. Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks. Our method maintains high accuracy, staying within ~5% of black-box performance while offering greater interpretability.

Related papers

• Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations [21.060815865649477]
  Vision-plus-Human-guided CBM (VH-CBM) is a hybrid approach that exploits both Vision-Language Models (VLMs) and a small amount of dense annotations.<n>VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data.
  arXiv  Detail & Related papers  (2026-05-13T10:07:11Z)
• Explaining CLIP Zero-shot Predictions Through Concepts [54.05282304471016]
  We introduce EZPC to explain CLIP's zero-shot predictions through human-understandable concepts.<n>Our method projects CLIP's joint image-text embeddings into a concept space learned from language descriptions.<n>Our approach maintains CLIP's strong zero-shot classification accuracy while providing meaningful concept-level explanations.
  arXiv  Detail & Related papers  (2026-03-30T09:31:33Z)
• Towards Faithful Multimodal Concept Bottleneck Models [0.07329200485567826]
  Concept Bottleneck Models (CBMs) are interpretable models that route predictions through a layer of human-interpretable concepts.<n>We introduce f-CBM, a multimodal CBM framework built on a vision-language backbone that jointly targets concept detection and leakage mitigation.<n>Experiments demonstrate that f-CBM achieves the best trade-off between task accuracy, concept detection, and leakage reduction.
  arXiv  Detail & Related papers  (2026-03-13T16:56:08Z)
• Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition [81.2779530670268]
  Vision-Language Models (VLMs) have significantly advanced zero-shot image recognition.<n>This paper enhances prompt by incorporating class-specific concepts.<n>Our method consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-03-09T03:11:11Z)
• Learning Concept Bottleneck Models from Mechanistic Explanations [4.071515891039002]
  Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision.<n>Mechanistic CBM builds the bottleneck directly from a black-box model's own learned concepts.<n>We show that M-CBMs consistently surpass prior CBMs at matched sparsity.
  arXiv  Detail & Related papers  (2026-03-07T21:09:01Z)
• Rethinking Concept Bottleneck Models: From Pitfalls to Solutions [53.84388497227224]
  Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts.<n>CBM-Suite is a methodological framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T19:37:49Z)
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts [79.18608192761512]
  Self-Explainable Models (SEMs) rely on Prototypical Concept Learning (PCL) to enable their visual recognition processes more interpretable.<n>We propose a Few-Shot Prototypical Concept Classification framework that mitigates two key challenges under low-data regimes: parametric imbalance and representation misalignment.<n>Our approach consistently outperforms existing SEMs by a notable margin, with 4.2%-8.7% relative gains in 5-way 5-shot classification.
  arXiv  Detail & Related papers  (2025-06-05T06:39:43Z)
• V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer [19.177297480709512]
  Concept Bottleneck Models (CBMs) offer inherent interpretability by translating images into human-comprehensible concepts.<n>Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks.<n>In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models.
  arXiv  Detail & Related papers  (2025-01-09T05:12:38Z)
• VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance [16.16577751549164]
  Concept Bottleneck Models (CBMs) provide interpretable prediction.<n>CBMs encode human-understandable concepts to explain models' decision.<n>We propose Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability.
  arXiv  Detail & Related papers  (2024-07-18T19:44:44Z)
• Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning [86.15009879251386]
  We propose a novel architecture and method of explainable classification with Concept Bottleneck Models (CBM) CBMs require an additional set of concepts to leverage. We show a significant increase in accuracy using sparse hidden layers in CLIP-based bottleneck models.
  arXiv  Detail & Related papers  (2024-04-04T09:43:43Z)
• M^2ConceptBase: A Fine-Grained Aligned Concept-Centric Multimodal Knowledge Base [61.53959791360333]
  We introduce M2ConceptBase, the first concept-centric multimodal knowledge base (MMKB)<n>We propose a context-aware approach to align concept-image and concept-description pairs using context information from image-text datasets.<n>Human studies confirm more than 95% alignment accuracy, underscoring its quality.
  arXiv  Detail & Related papers  (2023-12-16T11:06:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.