Caption Bottleneck Models

• URL: http://arxiv.org/abs/2607.00578v1
• Date: Wed, 01 Jul 2026 08:00:48 GMT
• Title: Caption Bottleneck Models
• Abstract summary: Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts.<n>We propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language.<n>Our results demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.
• Score: 7.606384988416701
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept Bottleneck Models (CBMs) provide interpretability by routing predictions through a layer of human-understandable concepts. However, defining an optimal concept set for a specific dataset remains an open challenge. Existing approaches rely on expensive expert annotations or LLM-generated lists based solely on class names. Even "open-vocabulary" variants typically depend on static concept sets, which restrict discovery and introduce label bias. Furthermore, traditional CBMs often suffer from information leakage, where unmodeled visual features bypass the bottleneck and compromise the integrity of the explanations. To overcome these limitations, we propose Caption Bottleneck Models (CaBM), a framework that circumvents the need for predefined concept sets by replacing rigid concept layers with free-form natural language. By representing images via LMM-generated captions and training a classifier strictly on this text, CaBM ensures a leakage-free architecture by construction. Additionally, by analyzing the text classifier post-training, CaBM autonomously discovers high-quality, dataset-specific concepts. Our results across fine- and coarse-grained benchmarks demonstrate that CaBM achieves competitive accuracy while preserving interpretability without the constraints of external dictionaries or manual labeling.

Related papers

• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• REC-CBM: Rubric-Aware Error-Correction Concept Bottleneck Models for Trustworthy Open-Ended Grading [11.935308398286361]
  We present REC-CBM, a rubric-aware error-correction concept bottleneck model for trustworthy open-ended grading.<n>It produces more faithful concept-level reasoning than both state-of-the-art baselines.<n>Overall, this work provides a practical, interpretable grading solution that enables educators to inspect, intervene in, and trust automated decisions.
  arXiv  Detail & Related papers  (2026-04-24T00:44:58Z)
• Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition [81.2779530670268]
  Vision-Language Models (VLMs) have significantly advanced zero-shot image recognition.<n>This paper enhances prompt by incorporating class-specific concepts.<n>Our method consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-03-09T03:11:11Z)
• Learning Concept Bottleneck Models from Mechanistic Explanations [4.071515891039002]
  Concept Bottleneck Models (CBMs) aim for ante-hoc interpretability by learning a bottleneck layer that predicts interpretable concepts before the decision.<n>Mechanistic CBM builds the bottleneck directly from a black-box model's own learned concepts.<n>We show that M-CBMs consistently surpass prior CBMs at matched sparsity.
  arXiv  Detail & Related papers  (2026-03-07T21:09:01Z)
• Rethinking Concept Bottleneck Models: From Pitfalls to Solutions [53.84388497227224]
  Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts.<n>CBM-Suite is a methodological framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T19:37:49Z)
• Concept-Aware Batch Sampling Improves Language-Image Pretraining [78.53540190580189]
  Concept-Aware Batch Sampling (CABS) is a simple yet effective batch sampling framework that flexibly constructs batches on-the-fly.<n>We show that CABS significantly benefits CLIP/SigLIP model classes and yields highly performant models.<n>Overall, CABS represents a strong open-source alternative to proprietary online data curation algorithms.
  arXiv  Detail & Related papers  (2025-11-25T18:58:07Z)
• Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability [54.420663939897686]
  We propose the Attribute-formed Language Bottleneck Model (ALBM) to achieve interpretable image recognition.<n>ALBM organizes concepts in the attribute-formed class-specific space, where concepts are descriptions of specific attributes for specific classes.<n>To further improve interpretability, we propose Visual Attribute Prompt Learning (VAPL) to extract visual features on fine-grained attributes.
  arXiv  Detail & Related papers  (2025-03-26T07:59:04Z)
• Towards Achieving Concept Completeness for Textual Concept Bottleneck Models [0.3694429692322631]
  This paper proposes a novel TCBM generator building concept labels in a fully unsupervised manner using a small language model.<n>CT-CBM iteratively targets and adds important and identifiable concepts in the bottleneck layer to create a complete concept basis.<n>CT-CBM achieves striking results against competitors in terms of concept basis and concept detection accuracy.
  arXiv  Detail & Related papers  (2025-02-16T12:28:43Z)
• Bayesian Concept Bottleneck Models with LLM Priors [8.895722261818209]
  Concept Bottleneck Models (CBMs) have been proposed as a compromise between white-box and black-box models, aiming to achieve interpretability without sacrificing accuracy.<n>This work investigates a novel approach that sidesteps these challenges: BC-LLM iteratively searches over a potentially infinite set of concepts within a Bayesian framework, in which Large Language Models (LLMs) serve as both a concept extraction mechanism and prior.
  arXiv  Detail & Related papers  (2024-10-21T01:00:33Z)
• Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts [8.028021897214238]
  "OpenCBM" is the first CBM with concepts of open vocabularies. Our model significantly outperforms the previous state-of-the-art CBM by 9% in the classification accuracy on the benchmark dataset CUB-200-2011.
  arXiv  Detail & Related papers  (2024-08-05T06:42:00Z)
• Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery [52.498055901649025]
  Concept Bottleneck Models (CBMs) have been proposed to address the 'black-box' problem of deep neural networks. We propose a novel CBM approach -- called Discover-then-Name-CBM (DN-CBM) -- that inverts the typical paradigm. Our concept extraction strategy is efficient, since it is agnostic to the downstream task, and uses concepts already known to the model.
  arXiv  Detail & Related papers  (2024-07-19T17:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.