Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing

• URL: http://arxiv.org/abs/2607.23368v1
• Date: Sat, 25 Jul 2026 21:17:57 GMT
• Title: Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing
• Abstract summary: We introduce ParseFIxLIP, an extension that incorporates the Tree-Gram Parsing interaction game used by FIxLIP.<n>Our smart_depth grouping strategy, merging tokens according to spaCy token dependency tree, successfully mitigates concept fragmentation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are demonstrating significant capabilities in medical tasks like radiology analysis, yet providing faithful and interpretable explanations remains a key consideration for their responsible deployment in clinical settings. However, existing explanation methods, such as the widely used FIxLIP framework, often struggle with the fine-grained nature of modern tokenizers. The tokenization problem fragments clinical concepts---splitting terms like "saddle embolus" into scattered, meaningless subwords---which leads to noisy, semantically incoherent cross-modal attributions. Such fragmentation also results in a combinatorial explosion of interaction possibilities, obscuring the model's true reasoning. To address this, we introduce ParseFIxLIP, an extension that incorporates the Tree-Gram Parsing into the Banzhaf interaction game used by FIxLIP. This semantically informed strategy utilizes dependency parsing trees to define explanation players by grouping related text tokens into semantically coherent units. Our smart_depth grouping strategy, merging tokens according to spaCy token dependency tree, successfully mitigates concept fragmentation, yielding substantially more interpretable cross-modal interactions by unifying complex medical concepts. Quantitatively, while baselines struggled with the high dimensionality of long captions, our parsing approach maintained statistical robustness and semantic parsimony. Qualitative analysis on BiomedCLIP, validated on medical imagery (ROCOv2) and general examples, confirms that the approach accurately captures the synergistic influence of grouped words on model predictions. In conclusion, our work offers intuitive and clinically relevant insights into VLM decision-making, fulfilling the critical need for coherent explanations in the medical domain.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• DrugReason: Dynamic Multi-View Reasoning over Knowledge Graph and Language Evidence for Drug Repurposing [55.45112712244231]
  DrugReason is a multi-view reasoning framework that integrates grounded KG reasoning with LLM-generated mechanistic inference for drug repurposing.<n> Experiments on PharmaDB, DDInter, and DrugBank show that DrugReason improves average performance over strong single-view reasoning baselines.
  arXiv  Detail & Related papers  (2026-09-06T18:54:26Z)
• SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation [5.487826115955642]
  Vision-language pre-training serves as a cornerstone for medical multimodal representation learning.<n>We propose textbfSCALPEL, a textbfSemantic textbfCross-modal textbfAlignment framework via textbfLLM-textbfPowered textbfEncoder textbfLearning.
  arXiv  Detail & Related papers  (2026-07-29T13:12:32Z)
• ReportMedSAM: Guiding Segmentation Through Radiology Reports [44.07905920142891]
  ReportMedSAM is a report-driven framework that replaces discrete extraction with a learnable concept bank.<n>Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms.<n>ReportMedSAM effectively interprets free-form reports, achieves competitive segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.
  arXiv  Detail & Related papers  (2026-05-08T13:32:15Z)
• DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis [14.035057926904559]
  Concept Bottleneck Models (CBMs) structure predictions through human-interpretable clinical concepts.<n>Existing CBMs typically overlook the contextual dependencies among concepts.<n>We propose an end-to-end interpretable framework emphDCG-Net that integrates multimodal alignment with structured concept reasoning.
  arXiv  Detail & Related papers  (2026-03-20T01:11:58Z)
• MIRNet: Integrating Constrained Graph-Based Reasoning with Pre-training for Diagnostic Medical Imaging [67.74482877175797]
  MIRNet is a novel framework that integrates self-supervised pre-training with constrained graph-based reasoning.<n>We introduce TongueAtlas-4K, a benchmark comprising 4,000 images annotated with 22 diagnostic labels.
  arXiv  Detail & Related papers  (2025-11-13T06:30:41Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• NEARL-CLIP: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding [51.63264715941068]
  textbfNEARL-CLIP (iunderlineNteracted quunderlineEry underlineAdaptation with ounderlineRthogonaunderlineL Regularization) is a novel cross-modality interaction VLM-based framework.
  arXiv  Detail & Related papers  (2025-08-06T05:44:01Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)
• Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions [5.763375492057694]
  This paper presents a multi-relational model that explicitly leverages such a structure to derive word embeddings from definitions. An empirical analysis demonstrates that the framework can help imposing the desired structural constraints. Experiments reveal the superiority of the Hyperbolic word embeddings over the Euclidean counterparts.
  arXiv  Detail & Related papers  (2023-05-12T08:16:06Z)
• Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning [24.215619918283462]
  We present a novel framework for learning medical visual representations directly from paired radiology reports. Our framework harnesses the naturally exhibited semantic correspondences between medical image and radiology reports at three different levels.
  arXiv  Detail & Related papers  (2022-10-12T09:31:39Z)
• Design considerations for a hierarchical semantic compositional framework for medical natural language understanding [3.7003326903946756]
  We describe a framework inspired by mechanisms of human cognition in an attempt to jump the NLP performance curve. The paper describes insights from four key aspects including semantic memory, semantic composition, semantic activation. We discuss the design of a generative semantic model and an associated semantic used to transform a free-text sentence into a logical representation of its meaning.
  arXiv  Detail & Related papers  (2022-04-05T09:04:34Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.