Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models

• URL: http://arxiv.org/abs/2607.27304v1
• Date: Wed, 29 Jul 2026 17:11:52 GMT
• Title: Position, Not Provenance: Separating Reasoning Mediation from Sycophancy in Medical Vision-Language Models
• Abstract summary: Medical vision-language models generate chain-of-thought reasoning before answering clinical questions.<n>We present CoT-Mediate, a behavioral framework that perturbs a single clinically meaningful attribute within a model's own generated reasoning.<n>We evaluate LLaVA-Med and MedGemma on 1,000 VQA-RAD samples each.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical vision-language models (VLMs) generate chain-of-thought (CoT) reasoning before answering clinical questions, but whether this reasoning causally influences predictions remains unclear. We present CoT-Mediate, a behavioral framework that perturbs a single clinically meaningful attribute within a model's own generated reasoning and measures whether the resulting prediction follows the edited reasoning. Our framework combines a dual-arm protocol comparing re-prompted evidence with prefix-forced continuation, together with a provenance-controlled intervention that varies only the attributed source of identical reasoning to disentangle reasoning mediation from sycophancy. We evaluate LLaVA-Med and MedGemma on 1,000 VQA-RAD samples each. Prefix-forced continuation consistently yields higher mediation faithfulness than re-prompting, while the provenance analysis reveals distinct model-specific deference behaviors. Across both models, removing visual evidence increases reliance on injected reasoning, whereas laterality is the least faithfully tracked clinical attribute. These results show that the mechanism used to inject reasoning substantially affects measured faithfulness and that contextual position, rather than stated provenance, is the primary determinant of whether medical VLMs use their generated reasoning.

Related papers

• Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models [47.16270570101431]
  Large Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns.<n>We develop a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified.<n>We propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport.
  arXiv  Detail & Related papers  (2026-05-19T17:46:40Z)
• Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs [68.58207076756237]
  We introduce ProCauEval, a perturbation-based evaluation protocol that shifts from outcome assessment to mechanism diagnosis.<n>We find that models faithfully perceive video content yet systematically underexploit it during causal reasoning.<n>We propose Anti-Distillation Policy Optimization (ADPO), a reinforcement learning framework built on negative teacher alignment.
  arXiv  Detail & Related papers  (2026-05-10T08:48:58Z)
• Dialectic-Med: Mitigating Diagnostic Hallucinations via Counterfactual Adversarial Multi-Agent Debate [2.645243404278707]
  We propose Dialectic-Med, a multi-agent framework that enforces diagnostic rigor through adversarial dialectics.<n>By explicitly modeling the cognitive process of falsification, our framework guarantees that diagnostic reasoning is tightly grounded in verified visual regions.
  arXiv  Detail & Related papers  (2026-04-13T10:09:36Z)
• Towards Responsible Multimodal Medical Reasoning via Context-Aligned Vision-Language Models [11.019570079187552]
  Medical vision-language models (VLMs) show strong performance on radiology tasks but often produce fluent yet weakly grounded conclusions.<n>We introduce a context-aligned reasoning framework that enforces agreement across heterogeneous clinical evidence before generating diagnostic conclusions.
  arXiv  Detail & Related papers  (2026-04-09T23:09:38Z)
• MedCausalX: Adaptive Causal Reasoning with Self-Reflection for Trustworthy Medical Vision-Language Models [10.466505116993451]
  Existing medical chain-of-thought models lack explicit mechanisms to represent and enforce causal reasoning.<n>We propose MedCausalX, an end-to-end framework explicitly models causal reasoning chains in medical VLMs.<n>We show that MedCausalX consistently outperforms state-of-the-art methods, improving diagnostic consistency by +5.4 points, reducing hallucination by over 10 points, and attaining top spatial grounding IoU.
  arXiv  Detail & Related papers  (2026-03-24T11:28:15Z)
• Faithful or Just Plausible? Evaluating the Faithfulness of Closed-Source LLMs in Medical Reasoning [0.0]
  We conduct a systematic evaluation of faithfulness in medical reasoning among three widely used closed-source large language models.<n>We find that CoT reasoning steps often do not causally drive predictions, and models readily incorporate external hints without acknowledgment.
  arXiv  Detail & Related papers  (2026-03-14T15:30:53Z)
• Making medical vision-language models think causally across modalities with retrieval-augmented cross-modal reasoning [16.243806723551454]
  Medical vision-language models (VLMs) achieve strong performance in diagnostic reporting and image-text alignment.<n>Their underlying reasoning mechanisms remain fundamentally correlational, exhibiting reliance on superficial statistical associations.<n>We propose Multimodal Causal Retrieval-Augmented Generation, a framework that integrates causal inference principles with multimodal retrieval.
  arXiv  Detail & Related papers  (2026-01-26T11:03:00Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Logical Satisfiability of Counterfactuals for Faithful Explanations in NLI [60.142926537264714]
  We introduce the methodology of Faithfulness-through-Counterfactuals. It generates a counterfactual hypothesis based on the logical predicates expressed in the explanation. It then evaluates if the model's prediction on the counterfactual is consistent with that expressed logic.
  arXiv  Detail & Related papers  (2022-05-25T03:40:59Z)
• Learning Causal Semantic Representation for Out-of-Distribution Prediction [125.38836464226092]
  We propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately. We show that CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error.
  arXiv  Detail & Related papers  (2020-11-03T13:16:05Z)
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder [52.139696854386976]
  The framework of variational autoencoder (VAE) is commonly used to disentangle independent factors from observations. We propose a new VAE based framework named CausalVAE, which includes a Causal Layer to transform independent factors into causal endogenous ones. Results show that the causal representations learned by CausalVAE are semantically interpretable, and their causal relationship as a Directed Acyclic Graph (DAG) is identified with good accuracy.
  arXiv  Detail & Related papers  (2020-04-18T20:09:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.