Concepts' Information Bottleneck Models

• URL: http://arxiv.org/abs/2602.14626v1
• Date: Mon, 16 Feb 2026 10:33:20 GMT
• Title: Concepts' Information Bottleneck Models
• Authors: Karim Galliamov, Syed M Ahsan Kazmi, Adil Khan, Adín Ramírez Rivera,
• Abstract summary: Concept Bottleneck Models (CBMs) aim to deliver interpretable predictions by routing decisions through a human-understandable concept layer.<n>We introduce an explicit Information Bottleneck regularizer on the concept layer that penalizes $I(X;C)$ while preserving task-relevant information in $I(C;Y)$, encouraging minimal-sufficient concept representations.
• Score: 9.435622803973898
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Concept Bottleneck Models (CBMs) aim to deliver interpretable predictions by routing decisions through a human-understandable concept layer, yet they often suffer reduced accuracy and concept leakage that undermines faithfulness. We introduce an explicit Information Bottleneck regularizer on the concept layer that penalizes $I(X;C)$ while preserving task-relevant information in $I(C;Y)$, encouraging minimal-sufficient concept representations. We derive two practical variants (a variational objective and an entropy-based surrogate) and integrate them into standard CBM training without architectural changes or additional supervision. Evaluated across six CBM families and three benchmarks, the IB-regularized models consistently outperform their vanilla counterparts. Information-plane analyses further corroborate the intended behavior. These results indicate that enforcing a minimal-sufficient concept bottleneck improves both predictive performance and the reliability of concept-level interventions. The proposed regularizer offers a theoretic-grounded, architecture-agnostic path to more faithful and intervenable CBMs, resolving prior evaluation inconsistencies by aligning training protocols and demonstrating robust gains across model families and datasets.

Related papers

• Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling [49.41422138354821]
  We propose a principled reward modeling framework that integrates non-negative factor analysis into the Bradley-Terry preference model.<n>BNRM represents rewards through a sparse, non-negative latent factor generative process.<n>We show that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.
  arXiv  Detail & Related papers  (2026-02-11T08:14:11Z)
• Controllable Concept Bottleneck Models [55.03639763625018]
  Controllable Concept Bottleneck Models (CCBMs)<n>CCBMs support three granularities of model editing: concept-label-level, concept-level, and data-level.<n>CCBMs enjoy mathematically rigorous closed-form approximations derived from influence functions that obviate the need for retraining.
  arXiv  Detail & Related papers  (2026-01-01T19:30:06Z)
• Towards more holistic interpretability: A lightweight disentangled Concept Bottleneck Model [5.700536552863068]
  Concept Bottleneck Models (CBMs) enhance interpretability by predicting human-understandable concepts as intermediate representations.<n>We propose a lightweight Disentangled Concept Bottleneck Model (LDCBM) that automatically groups visual features into semantically meaningful components.<n> Experiments on three diverse datasets demonstrate that LDCBM achieves higher concept and class accuracy, outperforming previous CBMs in both interpretability and classification performance.
  arXiv  Detail & Related papers  (2025-10-17T15:59:30Z)
• Post-hoc Stochastic Concept Bottleneck Models [18.935442650741]
  Concept Bottleneck Models (CBMs) are interpretable models that predict the target variable through high-level human-understandable concepts.<n>We introduce Post-hoc Concept Bottleneck Models (PSCBMs), a lightweight method that augments any pre-trained CBM with a normal distribution over concepts without retraining the backbone model.<n>We show that PSCBMs perform much better than CBMs under interventions, while remaining far more efficient than retraining a similar model from scratch.
  arXiv  Detail & Related papers  (2025-10-09T13:42:54Z)
• Interpretable Reward Modeling with Active Concept Bottlenecks [54.00085739303773]
  We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning.<n>Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts.<n>We formalize an active learning strategy that dynamically acquires the most informative concept labels.
  arXiv  Detail & Related papers  (2025-07-07T06:26:04Z)
• Interpretable Few-Shot Image Classification via Prototypical Concept-Guided Mixture of LoRA Experts [79.18608192761512]
  Self-Explainable Models (SEMs) rely on Prototypical Concept Learning (PCL) to enable their visual recognition processes more interpretable.<n>We propose a Few-Shot Prototypical Concept Classification framework that mitigates two key challenges under low-data regimes: parametric imbalance and representation misalignment.<n>Our approach consistently outperforms existing SEMs by a notable margin, with 4.2%-8.7% relative gains in 5-way 5-shot classification.
  arXiv  Detail & Related papers  (2025-06-05T06:39:43Z)
• Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization [8.368983263620889]
  Concept Bottleneck Models (CBMs) propose to enhance the trustworthiness of AI systems by constraining their decisions on a set of human-understandable concepts.<n>CBMs typically assume that datasets contain accurate concept labels, which can significantly degrade performance.<n>We introduce the Concept Preference Optimization (CPO) objective, which effectively mitigates the negative impact of concept mislabeling on CBM performance.
  arXiv  Detail & Related papers  (2025-04-25T02:43:10Z)
• Leakage and Interpretability in Concept-Based Models [0.24466725954625887]
  Concept Bottleneck Models aim to improve interpretability by predicting high-level intermediate concepts.<n>They are known to suffer from information leakage, whereby models exploit unintended information encoded within the learned concepts.<n>We introduce an information-theoretic framework to rigorously characterise and quantify leakage.
  arXiv  Detail & Related papers  (2025-04-18T22:21:06Z)
• MulCPred: Learning Multi-modal Concepts for Explainable Pedestrian Action Prediction [57.483718822429346]
  MulCPred is proposed that explains its predictions based on multi-modal concepts represented by training samples. MulCPred is evaluated on multiple datasets and tasks.
  arXiv  Detail & Related papers  (2024-09-14T14:15:28Z)
• Improving Intervention Efficacy via Concept Realignment in Concept Bottleneck Models [57.86303579812877]
  Concept Bottleneck Models (CBMs) ground image classification on human-understandable concepts to allow for interpretable model decisions. Existing approaches often require numerous human interventions per image to achieve strong performances. We introduce a trainable concept realignment intervention module, which leverages concept relations to realign concept assignments post-intervention.
  arXiv  Detail & Related papers  (2024-05-02T17:59:01Z)
• Benchmarking and Enhancing Disentanglement in Concept-Residual Models [4.177318966048984]
  Concept bottleneck models (CBMs) are interpretable models that first predict a set of semantically meaningful features. CBMs' performance depends on the engineered features and can severely suffer from incomplete sets of concepts. This work proposes three novel approaches to mitigate information leakage by disentangling concepts and residuals.
  arXiv  Detail & Related papers  (2023-11-30T21:07:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.