PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models

• URL: http://arxiv.org/abs/2602.21428v1
• Date: Tue, 24 Feb 2026 23:03:50 GMT
• Title: PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models
• Abstract summary: We introduce Paraphrase Sensitivity Failure (PSF)-Med, a benchmark of 19,748 chest Xray questions paired with about 92,000 paraphrases.<n>We measure yes/no flips for the same image and find flip rates from 8% to 58%.<n>We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts.
• Score: 2.064612766965483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical Vision Language Models (VLMs) can change their answers when clinicians rephrase the same question, which raises deployment risks. We introduce Paraphrase Sensitivity Failure (PSF)-Med, a benchmark of 19,748 chest Xray questions paired with about 92,000 meaningpreserving paraphrases across MIMIC-CXR and PadChest. Across six medical VLMs, we measure yes/no flips for the same image and find flip rates from 8% to 58%. However, low flip rate does not imply visual grounding: text-only baselines show that some models stay consistent even when the image is removed, suggesting they rely on language priors. To study mechanisms in one model, we apply GemmaScope 2 Sparse Autoencoders (SAEs) to MedGemma 4B and analyze FlipBank, a curated set of 158 flip cases. We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts. In causal patching, removing this feature's contribution recovers 45% of the yesminus-no logit margin on average and fully reverses 15% of flips. Acting on this finding, we show that clamping the identified feature at inference reduces flip rates by 31% relative with only a 1.3 percentage-point accuracy cost, while also decreasing text-prior reliance. These results suggest that flip rate alone is not enough; robustness evaluations should test both paraphrase stability and image reliance.

Related papers

• ModaLens: Measuring Image Sensitivity in Report-Conditioned Medical VLMs [3.1976608836761353]
  ModaLens measures how report availability changes image sensitivity.<n>The model's generated answer changes on 4.26 percent of trials with the report and 20.94 percent without it.<n>The original prompt with a lowercase first-token readout gives 4.70 percent against 17.07 percent, and substitutions also move continuous answer scores where the binary prediction does not change.
  arXiv  Detail & Related papers  (2026-09-14T14:25:20Z)
• DementiaCare-Bench: A Modality-Validated Video Benchmark [83.92703885588868]
  Dementia affects an estimated 57 million people worldwide.<n>For most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia.<n>We present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories.
  arXiv  Detail & Related papers  (2026-09-11T14:52:49Z)
• MedQA-MM: Shortcuts Behind Medical Visual Reasoning [45.333881843709804]
  A benchmark score credits final answers, but not the route by which an item can be answered.<n>In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding.<n>Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition.
  arXiv  Detail & Related papers  (2026-09-03T01:40:19Z)
• Consistent but Dangerous: Per-Sample Safety Classification Reveals False Reliability in Medical Vision-Language Models [2.064612766965483]
  We show a model can achieve perfect consistency by relying on text patterns rather than the input image.<n>We introduce a four-quadrant per-sample safety taxonomy that jointly evaluates consistency (stable predictions across paraphrased prompts) and image reliance.
  arXiv  Detail & Related papers  (2026-03-22T00:06:53Z)
• Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models [2.064612766965483]
  Medical Vision-Language Models can give different yes or no answers to rephrasings of the same clinical question.<n>We study this in MedGemma-4B using PSF-Med Sadanandan and Behzadan (2025), which provides paraphrase pairs for systematic consistency evaluation on medical VQA.<n>We fine-tune Low-Rank Adaptation (LoRA) adapters with a combined loss that balances paraphrase consistency with answer accuracy.
  arXiv  Detail & Related papers  (2026-02-24T23:30:50Z)
• Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations [19.488236277427358]
  Vision-language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process.<n>We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications.
  arXiv  Detail & Related papers  (2025-10-13T09:28:22Z)
• MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoning [7.167933033102407]
  We introduce MEDEQUALQA, a counterfactual benchmark that perturbs only patient pronouns while holding critical symptoms and conditions constant.<n>We evaluate a GPT-4.1 model and compute Semantic Textual Similarity (STS) between reasoning traces to measure stability across pronoun variants.<n>Our results show overall high similarity (mean STS >0.80), but reveal consistent localized divergences in cited risk factors, guideline anchors, and differential ordering.
  arXiv  Detail & Related papers  (2025-10-09T22:12:58Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• Calibrated and Robust Foundation Models for Vision-Language and Medical Image Tasks Under Distribution Shift [2.292525568003776]
  Foundation models like CLIP and SAM have advanced computer vision and medical imaging via low-shot transfer learning, aiding CADD with limited data.<n>We propose textbfStaRFM, a fusion of Fisher information penalty (FIP) and confidence misalignment penalty (CMP) tackling both challenges.<n>It applies FIP, extended to 3D via patch-wise regularization, to reduce embedding shift, and CMP, reformulated for voxel-level predictions, to segmentation uncertainty.
  arXiv  Detail & Related papers  (2025-07-12T09:39:07Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• MOZART: Ensembling Approach for COVID-19 Detection using Chest X-Ray Imagery [0.0]
  COVID-19, has led to a global pandemic that strained the healthcare systems. Traditional convolutional neural networks (CNNs) achieve impressive accuracy. We introduce the MOZART framework, an ensemble learning approach that enhances the virus detection.
  arXiv  Detail & Related papers  (2024-10-11T21:02:58Z)
• Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation [58.77994391566484]
  We propose W1KP, a human-calibrated measure of variability in a set of images. Our best perceptual distance outperforms nine baselines by up to 18 points in accuracy. We analyze 56 linguistic features of real prompts, finding that the prompt's length, CLIP embedding norm, concreteness, and word senses influence variability most.
  arXiv  Detail & Related papers  (2024-06-12T17:59:27Z)
• Automated SSIM Regression for Detection and Quantification of Motion Artefacts in Brain MR Images [54.739076152240024]
  Motion artefacts in magnetic resonance brain images are a crucial issue. The assessment of MR image quality is fundamental before proceeding with the clinical diagnosis. An automated image quality assessment based on the structural similarity index (SSIM) regression has been proposed here.
  arXiv  Detail & Related papers  (2022-06-14T10:16:54Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.