Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning

• URL: http://arxiv.org/abs/2603.08921v1
• Date: Mon, 09 Mar 2026 20:39:46 GMT
• Title: Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning
• Abstract summary: Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI.<n>We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models.<n>Our framework achieves interpretability and forms an end-to-end bridge from medical image analysis to decision-making.
• Score: 3.2723884637302443
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features to a set of meaningful concepts for task-specific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, discrete concept representations often overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multitask objective combining multimodal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. MedCBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments on non-medical datasets achieve 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• ConceptM$^3$oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology [33.27476666910628]
  ConceptM$3$oE embeds concept formation directly within interaction-aware mixture-of-experts pathways.<n>The architecture decomposes evidence into modality-specific, redundant, and synergistic experts, which are then projected into structured concept bottlenecks.<n>In data-limited regimes, ConceptM$3$oE improves limited-data performance, increasing macro-F1 from 56.41% to 66.70% at small training sizes.
  arXiv  Detail & Related papers  (2026-05-23T04:55:16Z)
• Clinical Cognition Alignment for Gastrointestinal Diagnosis with Multimodal LLMs [63.535652574541764]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable potential in medical image analysis.<n>Their application in gastrointestinal endoscopy is currently hindered by two critical limitations.<n>We propose a novel Clinical-Cognitive-Aligned (CogAlign) framework to address these challenges.
  arXiv  Detail & Related papers  (2026-03-21T07:47:37Z)
• DCG-Net: Dual Cross-Attention with Concept-Value Graph Reasoning for Interpretable Medical Diagnosis [14.035057926904559]
  Concept Bottleneck Models (CBMs) structure predictions through human-interpretable clinical concepts.<n>Existing CBMs typically overlook the contextual dependencies among concepts.<n>We propose an end-to-end interpretable framework emphDCG-Net that integrates multimodal alignment with structured concept reasoning.
  arXiv  Detail & Related papers  (2026-03-20T01:11:58Z)
• Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generation [12.226029763256962]
  Radiology Report Generation through Vision-Language Models (VLMs) promises to reduce documentation burden, improve reporting consistency, and accelerate clinical adoption.<n>Existing research treats interpretability and accuracy as separate objectives, with concept-based explainability techniques focusing primarily on transparency.<n>We present Concept-Enhanced Multimodal RAG (CEMRAG), a unified framework that decomposes visual representations into interpretable clinical concepts.
  arXiv  Detail & Related papers  (2026-02-17T15:18:07Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• RAD: Towards Trustworthy Retrieval-Augmented Multi-modal Clinical Diagnosis [56.373297358647655]
  Retrieval-Augmented Diagnosis (RAD) is a novel framework that injects external knowledge into multimodal models directly on downstream tasks.<n>RAD operates through three key mechanisms: retrieval and refinement of disease-centered knowledge from multiple medical sources, a guideline-enhanced contrastive loss transformer, and a dual decoder.
  arXiv  Detail & Related papers  (2025-09-24T10:36:14Z)
• Towards generating more interpretable counterfactuals via concept vectors: a preliminary study on chest X-rays [46.667021835430155]
  We map clinical concepts into the latent space of generative models to identify Concept Activation Vectors (CAVs)<n>The extracted concepts are stable across datasets, enabling visual explanations that highlight clinically relevant features.<n>Preliminary results on chest X-rays show promise for large pathologies like cardiomegaly, while smaller pathologies remain challenging.
  arXiv  Detail & Related papers  (2025-06-04T15:23:12Z)
• An Explainable Biomedical Foundation Model via Large-Scale Concept-Enhanced Vision-Language Pre-training [40.16314726875265]
  ConceptCLIP is the first explainable biomedical foundation model that achieves state-of-the-art diagnostic accuracy.<n>We develop ConceptCLIP through a novel dual-alignment approach that simultaneously learns global image-text representations and fine-grained region-concept associations.
  arXiv  Detail & Related papers  (2025-01-26T16:07:11Z)
• Integrating Clinical Knowledge into Concept Bottleneck Models [18.26357481872999]
  Concept bottleneck models (CBMs) predict human-interpretable concepts before predicting the final output. We propose integrating clinical knowledge to refine CBMs, better aligning them with clinicians' decision-making processes. We validate our approach on two datasets of medical images: white blood cell and skin images.
  arXiv  Detail & Related papers  (2024-07-09T07:03:42Z)
• Aligning Human Knowledge with Visual Concepts Towards Explainable Medical Image Classification [8.382606243533942]
  We introduce a simple yet effective framework, Explicd, towards Explainable language-informed criteria-based diagnosis. By leveraging a pretrained vision-language model, Explicd injects these criteria into the embedding space as knowledge anchors. The final diagnostic outcome is determined based on the similarity scores between the encoded visual concepts and the textual criteria embeddings.
  arXiv  Detail & Related papers  (2024-06-08T23:23:28Z)
• MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment [4.861768967055006]
  We propose a multi-modal explainable disease diagnosis framework that meticulously aligns medical images and clinical-related concepts semantically at multiple strata. Our method, while preserving model interpretability, attains high performance and label efficiency for concept detection and disease diagnosis.
  arXiv  Detail & Related papers  (2024-01-16T17:45:01Z)
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models [49.95603725998561]
  We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
  arXiv  Detail & Related papers  (2023-10-04T21:57:09Z)
• Interpretable Vertebral Fracture Diagnosis [69.68641439851777]
  Black-box neural network models learn clinically relevant features for fracture diagnosis. This work identifies the concepts networks use for vertebral fracture diagnosis in CT images.
  arXiv  Detail & Related papers  (2022-03-30T13:07:41Z)
• BI-RADS-Net: An Explainable Multitask Learning Approach for Cancer Diagnosis in Breast Ultrasound Images [69.41441138140895]
  This paper introduces BI-RADS-Net, a novel explainable deep learning approach for cancer detection in breast ultrasound images. The proposed approach incorporates tasks for explaining and classifying breast tumors, by learning feature representations relevant to clinical diagnosis. Explanations of the predictions (benign or malignant) are provided in terms of morphological features that are used by clinicians for diagnosis and reporting in medical practice.
  arXiv  Detail & Related papers  (2021-10-05T19:14:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.