Enhancing Concept Localization in CLIP-based Concept Bottleneck Models

• URL: http://arxiv.org/abs/2510.07115v1
• Date: Wed, 08 Oct 2025 15:07:16 GMT
• Title: Enhancing Concept Localization in CLIP-based Concept Bottleneck Models
• Authors: Rémi Kazmierczak, Steve Azzolin, Eloïse Berthier, Goran Frehse, Gianni Franchi,
• Abstract summary: We show that Concept Bottleneck Models (CBMs) do not require explicit concept annotations, relying instead on concepts extracted using CLIP in a zero-shot manner.<n>We introduce Concept Hallucination Inhibition via Localized Interpretability (CHILI), a technique that disentangles image embeddings and localizes pixels corresponding to target concepts.
• Score: 11.592826680892367
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper addresses explainable AI (XAI) through the lens of Concept Bottleneck Models (CBMs) that do not require explicit concept annotations, relying instead on concepts extracted using CLIP in a zero-shot manner. We show that CLIP, which is central in these techniques, is prone to concept hallucination, incorrectly predicting the presence or absence of concepts within an image in scenarios used in numerous CBMs, hence undermining the faithfulness of explanations. To mitigate this issue, we introduce Concept Hallucination Inhibition via Localized Interpretability (CHILI), a technique that disentangles image embeddings and localizes pixels corresponding to target concepts. Furthermore, our approach supports the generation of saliency-based explanations that are more interpretable.

Related papers

• ConceptPrism: Concept Disentanglement in Personalized Diffusion Models via Residual Token Optimization [11.472088067393074]
  ConceptPrism is a novel framework that automatically disentangles the shared visual concept from image-specific residuals.<n>In experiments, ConceptPrism effectively resolves concept entanglement, achieving a significantly improved trade-off between fidelity and alignment.
  arXiv  Detail & Related papers  (2026-02-23T07:46:19Z)
• Concepts from Representations: Post-hoc Concept Bottleneck Models via Sparse Decomposition of Visual Representations [20.859723044900154]
  This paper introduces PCBM-ReD, a novel pipeline that retrofits interpretability onto pretrained opaque models.<n>It achieves state-of-the-art accuracy, narrows the performance gap with end-to-end models, and exhibits better interpretability.
  arXiv  Detail & Related papers  (2026-01-18T08:01:44Z)
• FaCT: Faithful Concept Traces for Explaining Neural Network Decisions [56.796533084868884]
  Deep networks have shown remarkable performance across a wide range of tasks, yet getting a global concept-level understanding of how they function remains a key challenge.<n>We put emphasis on the faithfulness of concept-based explanations and propose a new model with model-inherent mechanistic concept-explanations.<n>Our concepts are shared across classes and, from any layer, their contribution to the logit and their input-visualization can be faithfully traced.
  arXiv  Detail & Related papers  (2025-10-29T13:35:46Z)
• CoPA: Hierarchical Concept Prompting and Aggregating Network for Explainable Diagnosis [8.56688324078793]
  Concept Prompting and Aggregating (CoPA) is a novel framework designed to capture multilayer concepts under prompt guidance.<n>Visual representations from each layer are aggregated to align with textual concept representations.<n>CoPA outperforms state-of-the-art methods on three public datasets.
  arXiv  Detail & Related papers  (2025-10-04T10:29:15Z)
• Post-Hoc Concept Disentanglement: From Correlated to Isolated Concept Representations [12.072112471560716]
  Concept Activation Vectors (CAVs) are widely used to model human-understandable concepts.<n>They are trained by identifying directions from the activations of concept samples to those of non-concept samples.<n>This method produces similar, non-orthogonal directions for correlated concepts, such as "beard" and "necktie"<n>This entanglement complicates the interpretation of concepts in isolation and can lead to undesired effects in CAV applications.
  arXiv  Detail & Related papers  (2025-03-07T15:45:43Z)
• V2C-CBM: Building Concept Bottlenecks with Vision-to-Concept Tokenizer [19.177297480709512]
  Concept Bottleneck Models (CBMs) offer inherent interpretability by translating images into human-comprehensible concepts.<n>Recent approaches have leveraged the knowledge of large language models to construct concept bottlenecks.<n>In this study, we investigate to avoid these issues by constructing CBMs directly from multimodal models.
  arXiv  Detail & Related papers  (2025-01-09T05:12:38Z)
• OmniPrism: Learning Disentangled Visual Concept for Image Generation [57.21097864811521]
  Creative visual concept generation often draws inspiration from specific concepts in a reference image to produce relevant outcomes.<n>We propose OmniPrism, a visual concept disentangling approach for creative image generation.<n>Our method learns disentangled concept representations guided by natural language and trains a diffusion model to incorporate these concepts.
  arXiv  Detail & Related papers  (2024-12-16T18:59:52Z)
• Non-confusing Generation of Customized Concepts in Diffusion Models [135.4385383284657]
  We tackle the common challenge of inter-concept visual confusion in compositional concept generation using text-guided diffusion models (TGDMs) Existing customized generation methods only focus on fine-tuning the second stage while overlooking the first one. We propose a simple yet effective solution called CLIF: contrastive image-language fine-tuning.
  arXiv  Detail & Related papers  (2024-05-11T05:01:53Z)
• Visual Concept-driven Image Generation with Text-to-Image Diffusion Model [65.96212844602866]
  Text-to-image (TTI) models have demonstrated impressive results in generating high-resolution images of complex scenes.<n>Recent approaches have extended these methods with personalization techniques that allow them to integrate user-illustrated concepts.<n>However, the ability to generate images with multiple interacting concepts, such as human subjects, as well as concepts that may be entangled in one, or across multiple, image illustrations remains illusive.<n>We propose a concept-driven TTI personalization framework that addresses these core challenges.
  arXiv  Detail & Related papers  (2024-02-18T07:28:37Z)
• Implicit Concept Removal of Diffusion Models [92.55152501707995]
  Text-to-image (T2I) diffusion models often inadvertently generate unwanted concepts such as watermarks and unsafe images. We present the Geom-Erasing, a novel concept removal method based on the geometric-driven control.
  arXiv  Detail & Related papers  (2023-10-09T17:13:10Z)
• Coarse-to-Fine Concept Bottleneck Models [9.910980079138206]
  This work targets ante hoc interpretability, and specifically Concept Bottleneck Models (CBMs) Our goal is to design a framework that admits a highly interpretable decision making process with respect to human understandable concepts, on two levels of granularity. Within this framework, concept information does not solely rely on the similarity between the whole image and general unstructured concepts; instead, we introduce the notion of concept hierarchy to uncover and exploit more granular concept information residing in patch-specific regions of the image scene.
  arXiv  Detail & Related papers  (2023-10-03T14:57:31Z)
• Concept Gradient: Concept-based Interpretation Without Linear Assumption [77.96338722483226]
  Concept Activation Vector (CAV) relies on learning a linear relation between some latent representation of a given model and concepts. We proposed Concept Gradient (CG), extending concept-based interpretation beyond linear concept functions. We demonstrated CG outperforms CAV in both toy examples and real world datasets.
  arXiv  Detail & Related papers  (2022-08-31T17:06:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.