Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs

• URL: http://arxiv.org/abs/2608.17427v1
• Date: Tue, 18 Aug 2026 06:45:08 GMT
• Title: Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMs
• Abstract summary: We propose Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST)<n>CAST operates entirely during inference and requires no manual annotations for anatomically grounded hallucination mitigation.<n>CAST consistently outperforms strong baselines and surpasses decoding strategies reliant on ground truth.
• Score: 21.705626227651624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical vision-language models (Med-VLMs) have demonstrated strong performance on medical visual question answering, yet they remain prone to hallucination, generating clinically unsupported statements that are insufficiently grounded in image evidence. Mitigation methods applied during decoding offer a practical solution, but they typically lack anatomical awareness or rely heavily on ground truth annotations, which limits their applicability. We propose Counterfactual Anatomy-guided Spatial-Temporal decoding (CAST), a framework that operates entirely during inference and requires no manual annotations for anatomically grounded hallucination mitigation. CAST automatically discovers anatomical regions relevant to the given query through broad medical segmentation. It then selects a compact, causally informative area using counterfactual intervention based on the drop in answer likelihood under occlusion. Guided by this chosen region, CAST performs a unified contrastive decoding process, combining classifier-free guidance to correct spatial attention with stepwise temporal contrast to regulate generation dynamics. Experiments on the SLAKE and MIMIC-CXR datasets across three Med-VLMs demonstrate that CAST consistently outperforms strong baselines and surpasses decoding strategies reliant on ground truth. Our results indicate that compact, automatically selected regions provide highly effective contrastive guidance without expert annotations, offering a practical and generalizable solution for improving spatial grounding and reducing hallucinations. Code is available at https://github.com/csyifan/CAST.

Related papers

• SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding [35.05885138023876]
  We propose SurgNarrator, a new generative retrieval framework tailored for surgical video understanding.<n>We construct a well-curated surgery-centric vocabulary from surgical captions to define a clinically meaningful retrieval space.<n>During inference, a hierarchical, procedure-aware retrieval strategy narrows the search space to the relevant procedure type, delivering fast and effective responses.
  arXiv  Detail & Related papers  (2026-08-05T10:39:19Z)
• Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence [45.78409219344429]
  Multimodal large language models (MLLMs) show strong promise for clinical VQA and radiology report generation, yet inference-time hallucinations undermine trustworthy use.<n>We propose a holistic, training-free evidence-injection framework that systematically mitigates hallucinations through dual-side evidence injection.<n>We conduct systematic evaluations on 2 tasks and 5 datasets using textttLLaVA-1.5-7B, texttLLaVA-Med-1.5-7B, texttQwen3-VL-8B/32B, and textttInternVL
  arXiv  Detail & Related papers  (2026-06-30T08:07:25Z)
• KEPIL: Knowledge-Enhanced Prompt-Image Learning for Prompt-Robust Disease Detection [6.447908430647854]
  radiological findings are long-tailed in practice, leaving some conditions underrepresented and making zero-shot inference essential.<n>We present textitKEPIL, a prompt-robust framework that integrates curated medical knowledge to stabilize zero-shot generalization.
  arXiv  Detail & Related papers  (2026-05-09T19:29:01Z)
• Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM [46.344226429758606]
  Segment Anything Model (SAM) offers powerful zero-shot capabilities, but collapses under the weak, generic, and noisy prompts that dominate real clinicals.<n>We introduce SPD, a Saliency-Guided Prompt Distillation framework that converts unreliable cues into robust guidance.<n>SPD consistently outperforms existing SAM adaptations and delivers large gains in both region-based and boundary-based metrics.
  arXiv  Detail & Related papers  (2026-04-25T14:09:11Z)
• Divide-then-Diagnose: Weaving Clinician-Inspired Contexts for Ultra-Long Capsule Endoscopy Videos [83.01543151239254]
  We define a new task, diagnosis-driven CE video summarization.<n>VideoCAP is the first CE dataset with diagnosis-driven annotations.<n>DiCE is a clinician-inspired framework that mirrors the standard CE reading workflow.
  arXiv  Detail & Related papers  (2026-04-23T16:07:51Z)
• MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling [51.31633278218137]
  MedScope is a tool-using clinical video reasoning model that performs coarse-to-fine evidence seeking over long-form procedures.<n>We build ClinVideoSuite, an evidence-centric, fine-grained clinical video suite.<n>On full and fine-grained video understanding benchmarks, MedScope achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-11T09:47:02Z)
• Anatomical Region-Guided Contrastive Decoding: A Plug-and-Play Strategy for Mitigating Hallucinations in Medical VLMs [20.507007953026346]
  Anatomical Region-Guided Contrastive Decoding (ARCD) is a plug-and-play strategy that mitigates hallucinations by providing targeted, region-specific guidance.<n>Our method is effective in improving regional understanding, reducing hallucinations, and enhancing overall diagnostic accuracy.
  arXiv  Detail & Related papers  (2025-12-19T03:11:20Z)
• Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation [61.350584471060756]
  Vision-grounded medical report generation aims to produce clinically accurate descriptions of medical images.<n>We propose Self-Supervised Anatomical Consistency Learning (SS-ACL) to align generated reports with corresponding anatomical regions.<n>SS-ACL constructs a hierarchical anatomical graph inspired by the invariant top-down inclusion structure of human anatomy.
  arXiv  Detail & Related papers  (2025-09-30T08:59:06Z)
• MambaEviScrib: Mamba and Evidence-Guided Consistency Enhance CNN Robustness for Scribble-Based Weakly Supervised Ultrasound Image Segmentation [15.766686386490234]
  Weakly supervised learning (WSL) based on sparse annotation has achieved encouraging performance.<n>This study attempts to introduce scribble-based WSL into ultrasound image segmentation tasks.<n>We propose leveraging predictions near decision boundaries effectively.
  arXiv  Detail & Related papers  (2024-09-28T14:50:45Z)
• Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding [60.89561661441736]
  We propose a novel method for few-shot medical image segmentation. We construct our few-shot image segmentor using a deep convolutional network trained episodically. We enhance discriminability of deep embedding to encourage clustering of the feature domains of the same class.
  arXiv  Detail & Related papers  (2020-12-10T04:01:07Z)
• Weakly-supervised Learning For Catheter Segmentation in 3D Frustum Ultrasound [74.22397862400177]
  We propose a novel Frustum ultrasound based catheter segmentation method. The proposed method achieved the state-of-the-art performance with an efficiency of 0.25 second per volume.
  arXiv  Detail & Related papers  (2020-10-19T13:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.