ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs

• URL: http://arxiv.org/abs/2609.15635v1
• Date: Mon, 14 Sep 2026 14:25:20 GMT
• Title: ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs
• Abstract summary: ModaLens measures how report availability changes image sensitivity.<n>The model's generated answer changes on 4.26 percent of trials with the report and 20.94 percent without it.<n>The original prompt with a lowercase first-token readout gives 4.70 percent against 17.07 percent, and substitutions also move continuous answer scores where the binary prediction does not change.
• Score: 3.1976608836761353
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A radiology report can already answer a clinical question, so it is hard to tell whether a vision-language model also uses the image. ModaLens, a paired image-swap audit, measures how report availability changes image sensitivity: MedGemma-27B on 3,199 paired MIMIC-CXR cases from 293 patients, all 14 questions per case (13 finding-specific and one composite), each image replaced by one from another study, usually of the same patient, with question and report fixed. Under an explicit answer instruction, the model's generated answer changes on 4.26 percent of trials with the report and 20.94 percent without it, a paired increase of 16.7 points (patient-clustered 95 percent CI 15.6 to 17.7), so report availability reduces image-swap sensitivity under this protocol; the original prompt with a lowercase first-token readout gives 4.70 percent against 17.07 percent, and substitutions also move continuous answer scores where the binary prediction does not change. The labels are derived from reports, which limits conclusions about visual correctness; the direction replicates in two further model lineages. Code, the exact prompts and a run record for every number are at https://github.com/criticaldata/MODALENS.

Related papers

• DementiaCare-Bench: A Modality-Validated Video Benchmark [83.92703885588868]
  Dementia affects an estimated 57 million people worldwide.<n>For most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia.<n>We present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories.
  arXiv  Detail & Related papers  (2026-09-11T14:52:49Z)
• MedQA-MM: Shortcuts Behind Medical Visual Reasoning [45.333881843709804]
  A benchmark score credits final answers, but not the route by which an item can be answered.<n>In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding.<n>Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition.
  arXiv  Detail & Related papers  (2026-09-03T01:40:19Z)
• MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models [2.2090506971647144]
  Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging.<n>Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image.<n>We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation.
  arXiv  Detail & Related papers  (2026-08-25T06:28:17Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Cross-modal linkage risk in clinical vision-language models [1.828619151598757]
  Vision-language models (VLMs) trained on paired chest radiographs and radiology reports learn a shared embedding space that can preserve instance-level image-report correspondence.<n>This poses a privacy risk in settings where radiographs and reports are deliberately kept separate after acquisition.<n>We formalized this as image-to-report retrieval and used public paired cohorts to audit the risk rather than as the privacy scenario.
  arXiv  Detail & Related papers  (2026-06-01T14:01:46Z)
• When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models [2.064612766965483]
  We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions)<n>Our experiments reveal several concerning findings.
  arXiv  Detail & Related papers  (2026-03-26T23:04:20Z)
• PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models [2.064612766965483]
  We introduce Paraphrase Sensitivity Failure (PSF)-Med, a benchmark of 19,748 chest Xray questions paired with about 92,000 paraphrases.<n>We measure yes/no flips for the same image and find flip rates from 8% to 58%.<n>We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts.
  arXiv  Detail & Related papers  (2026-02-24T23:03:50Z)
• Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations [19.488236277427358]
  Vision-language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process.<n>We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications.
  arXiv  Detail & Related papers  (2025-10-13T09:28:22Z)
• Contrastive Learning with Counterfactual Explanations for Radiology Report Generation [83.30609465252441]
  We propose a textbfCountertextbfFactual textbfExplanations-based framework (CoFE) for radiology report generation. Counterfactual explanations serve as a potent tool for understanding how decisions made by algorithms can be changed by asking what if'' scenarios. Experiments on two benchmarks demonstrate that leveraging the counterfactual explanations enables CoFE to generate semantically coherent and factually complete reports.
  arXiv  Detail & Related papers  (2024-07-19T17:24:25Z)
• Multimodal Image-Text Matching Improves Retrieval-based Chest X-Ray Report Generation [3.6664023341224827]
  Contrastive X-Ray REport Match (X-REM) is a novel retrieval-based radiology report generation module. X-REM uses an image-text matching score to measure the similarity of a chest X-ray image and radiology report for report retrieval.
  arXiv  Detail & Related papers  (2023-03-29T04:00:47Z)
• DeltaNet:Conditional Medical Report Generation for COVID-19 Diagnosis [54.93879264615525]
  We propose DeltaNet to generate medical reports automatically. DeltaNet employs three steps to generate a report. We evaluate DeltaNet on a COVID-19 dataset, where DeltaNet outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-11-12T07:41:03Z)
• Automated SSIM Regression for Detection and Quantification of Motion Artefacts in Brain MR Images [54.739076152240024]
  Motion artefacts in magnetic resonance brain images are a crucial issue. The assessment of MR image quality is fundamental before proceeding with the clinical diagnosis. An automated image quality assessment based on the structural similarity index (SSIM) regression has been proposed here.
  arXiv  Detail & Related papers  (2022-06-14T10:16:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.