If Concept Bottlenecks are the Question, are Foundation Models the Answer?

• URL: http://arxiv.org/abs/2504.19774v2
• Date: Tue, 29 Apr 2025 10:54:44 GMT
• Title: If Concept Bottlenecks are the Question, are Foundation Models the Answer?
• Authors: Nicola Debole, Pietro Barbiero, Francesco Giannini, Andrea Passerini, Stefano Teso, Emanuele Marconato,
• Abstract summary: Concept Bottleneck Models (CBMs) are neural networks designed to conjoin high performance with ante-hoc interpretability.<n>"VLM-CBM" architectures replace manual annotations with weak supervision from foundation models.<n>We put state-of-the-art VLM-CBMs to the test, analyzing their learned concepts empirically using a selection of significant metrics.
• Score: 20.91927788087174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept Bottleneck Models (CBMs) are neural networks designed to conjoin high performance with ante-hoc interpretability. CBMs work by first mapping inputs (e.g., images) to high-level concepts (e.g., visible objects and their properties) and then use these to solve a downstream task (e.g., tagging or scoring an image) in an interpretable manner. Their performance and interpretability, however, hinge on the quality of the concepts they learn. The go-to strategy for ensuring good quality concepts is to leverage expert annotations, which are expensive to collect and seldom available in applications. Researchers have recently addressed this issue by introducing "VLM-CBM" architectures that replace manual annotations with weak supervision from foundation models. It is however unclear what is the impact of doing so on the quality of the learned concepts. To answer this question, we put state-of-the-art VLM-CBMs to the test, analyzing their learned concepts empirically using a selection of significant metrics. Our results show that, depending on the task, VLM supervision can sensibly differ from expert annotations, and that concept accuracy and quality are not strongly correlated. Our code is available at https://github.com/debryu/CQA.

Related papers

• Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding [58.364933651703524]
  We show that concentrated massive values consistently emerge in specific regions of attention queries.<n>These massive values play a critical role in interpreting contextual knowledge.<n>We trace the emergence of massive values and find that such concentration is caused by Rotary Positional.
  arXiv  Detail & Related papers  (2025-02-03T17:47:03Z)
• CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification [8.470147509053819]
  Concept Bottleneck Models (CBMs) tackle the latter by constraining the final disease prediction on a set of predefined and human-interpretable concepts. We propose a simple, yet effective, methodology, CBVLM, which tackles both of the aforementioned challenges. By grounding the final diagnosis on the predicted concepts, we ensure explainability, and by leveraging the few-shot capabilities of LVLMs, we drastically lower the annotation cost.
  arXiv  Detail & Related papers  (2025-01-21T16:38:04Z)
• DCBM: Data-Efficient Visual Concept Bottleneck Models [13.36057999450821]
  Concept Bottleneck Models (CBMs) enhance interpretability of neural networks by basing predictions on human-understandable concepts.<n>We propose Data-efficient CBMs, which reduce the need for large sample sizes during concept generation while preserving interpretability.
  arXiv  Detail & Related papers  (2024-12-16T09:04:58Z)
• Response Wide Shut: Surprising Observations in Basic Vision Language Model Capabilities [30.176918208200604]
  Vision-Language Models (VLMs) have emerged as general purpose tools for addressing a variety of complex computer vision problems. These models have been shown to be highly capable, but also lacking some basic visual understanding skills. This paper sets out to understand the limitations of SoTA VLMs on fundamental visual tasks.
  arXiv  Detail & Related papers  (2024-08-13T08:26:32Z)
• Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery [52.498055901649025]
  Concept Bottleneck Models (CBMs) have been proposed to address the 'black-box' problem of deep neural networks. We propose a novel CBM approach -- called Discover-then-Name-CBM (DN-CBM) -- that inverts the typical paradigm. Our concept extraction strategy is efficient, since it is agnostic to the downstream task, and uses concepts already known to the model.
  arXiv  Detail & Related papers  (2024-07-19T17:50:11Z)
• Self-supervised Interpretable Concept-based Models for Text Classification [9.340843984411137]
  This paper proposes a self-supervised Interpretable Concept Embedding Models (ICEMs) We leverage the generalization abilities of Large-Language Models to predict the concepts labels in a self-supervised way. ICEMs can be trained in a self-supervised way achieving similar performance to fully supervised concept-based models and end-to-end black-box ones.
  arXiv  Detail & Related papers  (2024-06-20T14:04:53Z)
• Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency [2.7719338074999547]
  Concept bottleneck models (CBMs) have emerged as critical tools in domains where interpretability is paramount. This study proposes underlinetextbfConceptual underlinetextbfLbedding via underlinetextbfEmbedding underlinetextbfApproximations for underlinetextbfReinforcing Interpretability and Transparency.
  arXiv  Detail & Related papers  (2024-06-13T06:04:34Z)
• Interpretable Prognostics with Concept Bottleneck Models [5.939858158928473]
  Concept Bottleneck Models (CBMs) are inherently interpretable neural network architectures based on concept explanations. CBMs enable domain experts to intervene on the concept activations at test-time. Our case studies demonstrate that the performance of CBMs can be on par or superior to black-box models.
  arXiv  Detail & Related papers  (2024-05-27T18:15:40Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation. We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
  arXiv  Detail & Related papers  (2023-11-08T20:41:18Z)
• Dynamic Clue Bottlenecks: Towards Interpretable-by-Design Visual Question Answering [58.64831511644917]
  We introduce an interpretable by design model that factors model decisions into intermediate human-legible explanations. We show that our inherently interpretable system can improve 4.64% over a comparable black-box system in reasoning-focused questions.
  arXiv  Detail & Related papers  (2023-05-24T08:33:15Z)
• Explainable fetal ultrasound quality assessment with progressive concept bottleneck models [6.734637459963132]
  We propose a holistic and explainable method for fetal ultrasound quality assessment.<n>We introduce human-readable concepts" into the task and imitate the sequential expert decision-making process.<n> Experiments show that our model outperforms equivalent concept-free models on an in-house dataset.
  arXiv  Detail & Related papers  (2022-11-19T09:31:19Z)
• CONVIQT: Contrastive Video Quality Estimator [63.749184706461826]
  Perceptual video quality assessment (VQA) is an integral component of many streaming and video sharing platforms. Here we consider the problem of learning perceptually relevant video quality representations in a self-supervised manner. Our results indicate that compelling representations with perceptual bearing can be obtained using self-supervised learning.
  arXiv  Detail & Related papers  (2022-06-29T15:22:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.