Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations

• URL: http://arxiv.org/abs/2605.16405v1
• Date: Wed, 13 May 2026 10:07:11 GMT
• Title: Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
• Abstract summary: Vision-plus-Human-guided CBM (VH-CBM) is a hybrid approach that exploits both Vision-Language Models (VLMs) and a small amount of dense annotations.<n>VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data.
• Score: 21.060815865649477
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Concept-bottleneck models (CBMs) are neural classifiers that compute predictions from high-level concepts extracted from the input. CBMs ensure stakeholders can understand the concepts -- and the predictions they entail -- by learning these from concept-level annotations, which are however seldom available. Recent CBM architectures work around this issue by obtaining annotations from Vision-Language Models (VLMs). While greatly broadening applicability, doing so can yield lower quality concepts and therefore less interpretable models. We strike for a middle ground by introducing Vision-plus-Human-guided CBM (VH-CBM), a hybrid approach that exploits both VLMs and a small amount of dense annotations. VH-CBM employs a Gaussian Process in the VLM's embedding space, which captures useful global information about the target domain, to propagate the expert's supervision to any target data point. Our empirical evaluation shows how VH-CBM predicts more accurate concepts than VLM-guided CBMs even when annotating as little as 1% of the data, while sporting better concept calibration and supporting active learning.

Related papers

• Multimodal Concept Bottleneck Models [19.41992921538502]
  Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.<n>We propose Multimodal Concept Bottleneck Model (MM-CBM) to address these issues and extend CBMs into CLIP.<n>Compared to existing methods, MM-CBM achieves up to 51.26% accuracy improvement on average across four standard benchmarks.
  arXiv  Detail & Related papers  (2026-06-18T07:41:50Z)
• Learning Concept Bottleneck Models from Mechanistic Explanations [4.071515891039002]
  Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision.<n>Mechanistic CBM builds the bottleneck directly from a black-box model's own learned concepts.<n>We show that M-CBMs consistently surpass prior CBMs at matched sparsity.
  arXiv  Detail & Related papers  (2026-03-07T21:09:01Z)
• Rethinking Concept Bottleneck Models: From Pitfalls to Solutions [53.84388497227224]
  Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts.<n>CBM-Suite is a methodological framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T19:37:49Z)
• Controllable Concept Bottleneck Models [55.03639763625018]
  Controllable Concept Bottleneck Models (CCBMs)<n>CCBMs support three granularities of model editing: concept-label-level, concept-level, and data-level.<n>CCBMs enjoy mathematically rigorous closed-form approximations derived from influence functions that obviate the need for retraining.
  arXiv  Detail & Related papers  (2026-01-01T19:30:06Z)
• AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models [63.05306474002547]
  Regulatory frameworks mandating the 'right to be forgotten' drive the need for machine unlearning.<n>We introduce AUVIC, a novel visual concept unlearning framework for MLLMs.<n>We show that AUVIC achieves state-of-the-art target forgetting rates while incurs minimal performance degradation on non-target concepts.
  arXiv  Detail & Related papers  (2025-11-14T13:35:32Z)
• Towards more holistic interpretability: A lightweight disentangled Concept Bottleneck Model [5.700536552863068]
  Concept Bottleneck Models (CBMs) enhance interpretability by predicting human-understandable concepts as intermediate representations.<n>We propose a lightweight Disentangled Concept Bottleneck Model (LDCBM) that automatically groups visual features into semantically meaningful components.<n> Experiments on three diverse datasets demonstrate that LDCBM achieves higher concept and class accuracy, outperforming previous CBMs in both interpretability and classification performance.
  arXiv  Detail & Related papers  (2025-10-17T15:59:30Z)
• Interpretable Reward Modeling with Active Concept Bottlenecks [54.00085739303773]
  We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning.<n>Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts.<n>We formalize an active learning strategy that dynamically acquires the most informative concept labels.
  arXiv  Detail & Related papers  (2025-07-07T06:26:04Z)
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts [79.18608192761512]
  Self-Explainable Models (SEMs) rely on Prototypical Concept Learning (PCL) to enable their visual recognition processes more interpretable.<n>We propose a Few-Shot Prototypical Concept Classification framework that mitigates two key challenges under low-data regimes: parametric imbalance and representation misalignment.<n>Our approach consistently outperforms existing SEMs by a notable margin, with 4.2%-8.7% relative gains in 5-way 5-shot classification.
  arXiv  Detail & Related papers  (2025-06-05T06:39:43Z)
• VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance [16.16577751549164]
  Concept Bottleneck Models (CBMs) provide interpretable prediction.<n>CBMs encode human-understandable concepts to explain models' decision.<n>We propose Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability.
  arXiv  Detail & Related papers  (2024-07-18T19:44:44Z)
• Auxiliary Losses for Learning Generalizable Concept-based Models [5.4066453042367435]
  Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human-understandable high-level concepts. We propose cooperative-Concept Bottleneck Model (coop-CBM) to overcome the performance trade-off.
  arXiv  Detail & Related papers  (2023-11-18T15:50:07Z)
• Evaluating and Explaining Large Language Models for Code Using Syntactic Structures [74.93762031957883]
  This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes. We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
  arXiv  Detail & Related papers  (2023-08-07T18:50:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.