Bridging Vision and Language Concepts through Optimal Transport Semantic Flow

• URL: http://arxiv.org/abs/2606.26891v1
• Date: Thu, 25 Jun 2026 11:24:44 GMT
• Title: Bridging Vision and Language Concepts through Optimal Transport Semantic Flow
• Abstract summary: Concept Bottleneck Models (CBMs) promise transparent reasoning by predicting through human-interpretable concepts.<n>Existing vision-language CBMs often rely on pre-aligned encoders or global cosine similarity.<n>We propose the Optimal Transport Flow Concept Bottleneck Model (OTF-CBM)
• Score: 16.961028982058732
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept Bottleneck Models (CBMs) promise transparent reasoning by predicting through human-interpretable concepts, yet their effectiveness fundamentally depends on how well visual and textual representations are aligned or matched. Existing vision-language CBMs often rely on pre-aligned encoders or global cosine similarity, which obscures fine-grained concept localization and fails to reflect true semantic geometry. In this work, we rethink concept alignment as a dynamic cross-modal transport process instead of static projection and propose the Optimal Transport Flow Concept Bottleneck Model (OTF-CBM). It first learns a data-driven semantic cost via Inverse Optimal Transport to measure cross-modal distances, and then performs unbalanced optimal-transport-based flow matching to model semantic transitions between visual patches and textual concepts. With velocity-based concept activation, OTF-CBM captures interpretable geometric relations without ODE integration. Experiments further show that OTF-CBM achieves superior classification accuracy and concept faithfulness, offering a new geometric and dynamical perspective for interpretable cross-modal reasoning.

Related papers

• MUtE: A Dual Framework for Concept Erasure and Counterfactual Interventions [0.0]
  Erasing concept-specific information from representations has been proven useful for mitigating bias or interpreting model decisions.<n>We revisit the optimal bounds of concept erasure to derive a novel class of erasure functions that naturally induce a deterministic, dual counterfactual mapping.
  arXiv  Detail & Related papers  (2026-09-10T08:47:11Z)
• Concept-wise Attention for Fine-grained Concept Bottleneck Models [8.484128495313607]
  Concept-wise Attention for Fine-grained Concept Bottleneck Models (CoAt-CBM) is a novel framework that achieves adaptive fine-grained image-concept alignment and high interpretability.<n>CoAt-CBM employs learnable concept-wise visual queries to adaptively obtain fine-grained concept-wise visual embeddings, which are then used to produce a concept score vector.
  arXiv  Detail & Related papers  (2026-04-17T06:43:30Z)
• Perception-Aware Multimodal Spatial Reasoning from Monocular Images [57.42071289037214]
  spatial reasoning from monocular images is essential for autonomous driving.<n>Current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception.<n>We propose a perception-aware multimodal reasoning framework that equipsVLMs with explicit object-centric grounding ability.
  arXiv  Detail & Related papers  (2026-03-07T02:05:12Z)
• Rethinking Concept Bottleneck Models: From Pitfalls to Solutions [53.84388497227224]
  Concept Bottleneck Models (CBMs) ground predictions in human-understandable concepts.<n>CBM-Suite is a methodological framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-05T19:37:49Z)
• Towards more holistic interpretability: A lightweight disentangled Concept Bottleneck Model [5.700536552863068]
  Concept Bottleneck Models (CBMs) enhance interpretability by predicting human-understandable concepts as intermediate representations.<n>We propose a lightweight Disentangled Concept Bottleneck Model (LDCBM) that automatically groups visual features into semantically meaningful components.<n> Experiments on three diverse datasets demonstrate that LDCBM achieves higher concept and class accuracy, outperforming previous CBMs in both interpretability and classification performance.
  arXiv  Detail & Related papers  (2025-10-17T15:59:30Z)
• Interpretable Reward Modeling with Active Concept Bottlenecks [54.00085739303773]
  We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning.<n>Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts.<n>We formalize an active learning strategy that dynamically acquires the most informative concept labels.
  arXiv  Detail & Related papers  (2025-07-07T06:26:04Z)
• Discovering Fine-Grained Visual-Concept Relations by Disentangled Optimal Transport Concept Bottleneck Models [16.617257464664572]
  Concept Bottleneck Models (CBMs) try to make the decision-making process transparent by exploring an intermediate concept space between the input image and the output prediction.<n>Existing CBMs just learn coarse-grained relations between the whole image and the concepts, less considering local image information.<n>This paper proposes a Disentangled Optimal Transport CBM framework to explore fine-grained visual-concept relations between local image patches and concepts.
  arXiv  Detail & Related papers  (2025-05-12T03:31:57Z)
• Improving Intervention Efficacy via Concept Realignment in Concept Bottleneck Models [57.86303579812877]
  Concept Bottleneck Models (CBMs) ground image classification on human-understandable concepts to allow for interpretable model decisions. Existing approaches often require numerous human interventions per image to achieve strong performances. We introduce a trainable concept realignment intervention module, which leverages concept relations to realign concept assignments post-intervention.
  arXiv  Detail & Related papers  (2024-05-02T17:59:01Z)
• Incremental Residual Concept Bottleneck Models [29.388549499546556]
  Concept Bottleneck Models (CBMs) map the black-box visual representations extracted by deep neural networks onto a set of interpretable concepts. We propose the Incremental Residual Concept Bottleneck Model (Res-CBM) to address the challenge of concept completeness. Our approach can be applied to any user-defined concept bank, as a post-hoc processing method to enhance the performance of any CBMs.
  arXiv  Detail & Related papers  (2024-04-13T12:02:19Z)
• Reasoning with the Theory of Mind for Pragmatic Semantic Communication [62.87895431431273]
  A pragmatic semantic communication framework is proposed in this paper. It enables effective goal-oriented information sharing between two-intelligent agents. Numerical evaluations demonstrate the framework's ability to achieve efficient communication with a reduced amount of bits.
  arXiv  Detail & Related papers  (2023-11-30T03:36:19Z)
• Modeling Temporal Concept Receptive Field Dynamically for Untrimmed Video Analysis [105.06166692486674]
  We study temporal concept receptive field of concept-based event representation. We introduce temporal dynamic convolution (TDC) to give stronger flexibility to concept-based event analytics. Different coefficients can generate appropriate and accurate temporal concept receptive field size according to input videos.
  arXiv  Detail & Related papers  (2021-11-23T04:59:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.