Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models

• URL: http://arxiv.org/abs/2603.00148v1
• Date: Tue, 24 Feb 2026 23:30:50 GMT
• Title: Mechanistically Guided LoRA Improves Paraphrase Consistency in Medical Vision-Language Models
• Abstract summary: Medical Vision-Language Models can give different yes or no answers to rephrasings of the same clinical question.<n>We study this in MedGemma-4B using PSF-Med Sadanandan and Behzadan (2025), which provides paraphrase pairs for systematic consistency evaluation on medical VQA.<n>We fine-tune Low-Rank Adaptation (LoRA) adapters with a combined loss that balances paraphrase consistency with answer accuracy.
• Score: 2.064612766965483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical Vision-Language Models can give different yes or no answers to rephrasings of the same clinical question. We study this in MedGemma-4B using PSF-Med Sadanandan and Behzadan (2025), which provides paraphrase pairs for systematic consistency evaluation on medical VQA. On MIMIC-CXR binary questions (n = 158), the baseline flip rate is 14.6% and mean margin difference is 1.63 logits. We validate that Gemma Scope 2 Sparse Autoencoders (SAEs) transfer to MedGemma activations, achieving R2 ~= 0.997 on both medical and general text (n = 100 prompts each, p < 0.001 for exceeding a 0.95 threshold). We then fine-tune Low-Rank Adaptation (LoRA) adapters with a combined loss that balances paraphrase consistency with answer accuracy. This combined approach prevents mode collapse that occurs with pure consistency training while reducing flip rate from 14.6% to 4.4% (p = 0.002, two-proportion z-test) and margin difference from 1.63 to 0.33 (79.5% reduction). Accuracy remains stable at 84.2% baseline versus 82.3% after training (-1.9pp, not significant). On PadChest Balanced (n = 250), flip rate drops from 13.6% to 7.8%, mean margin difference drops from 1.08 to 0.35 (67.9% reduction), and accuracy increases from 66.4% to 69.4%. A layer-range ablation shows that early layers reduce margin differences more than mechanistically selected middle layers.

Related papers

• Asymmetric Focal Loss Improves Graph Neural Network Prediction of Drug-Drug Interactions [0.0]
  Graph neural networks improve prediction of polypharmacy side effects.<n>Standard binary cross-entropy training allocates equal capacity to well-classified and difficult examples.<n>We evaluated whether an asymmetric focal objective could improve multi-relational drug-drug interaction prediction.
  arXiv  Detail & Related papers  (2026-07-08T16:28:57Z)
• Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering [4.896659944013307]
  BiRG-LoRA is a single-adapter rank-gated LoRA method for medical question answering.<n>BiRG-LoRA achieves the highest four-benchmark macro-average accuracy among trainable PEFT baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:59:05Z)
• LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data [2.1443570696048906]
  Large language models (LLMs) are increasingly applied to structured clinical data.<n>We compare Qwen 2.5 7B and XGBoost on a prediction task via attribution divergence analysis.
  arXiv  Detail & Related papers  (2026-06-17T18:49:44Z)
• Measuring Epistemic Resilience of LLMs Under Misleading Medical Context [47.20527783144983]
  Large language models (LLMs) now reach expert-level scores on medical licensing exams.<n>We show this assumption is fragile: when misleading context is injected into questions that LLMs originally answer correctly, they abandon the correct answer.<n>We call the ability to maintain correct judgment under adversarial context epistemic resilience, and introduce MedMisBench to measure it.
  arXiv  Detail & Related papers  (2026-06-10T16:27:26Z)
• A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks [46.43161388634735]
  Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years.<n>We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families.
  arXiv  Detail & Related papers  (2026-06-08T18:40:10Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Decomposing Physician Disagreement in HealthBench [0.0687531213383208]
  We decompose physician disagreement in the HealthBench medical AI evaluation dataset to understand where variance resides and what observable features can explain it.<n>The dominant 81.8% case-level residual is not reduced by HealthBench's metadata labels.<n>Disagreement follows an inverted-U with completion quality, confirming physicians agree on clearly good or bad outputs but split on borderline cases.
  arXiv  Detail & Related papers  (2026-02-26T08:47:42Z)
• PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models [2.064612766965483]
  We introduce Paraphrase Sensitivity Failure (PSF)-Med, a benchmark of 19,748 chest Xray questions paired with about 92,000 paraphrases.<n>We measure yes/no flips for the same image and find flip rates from 8% to 58%.<n>We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts.
  arXiv  Detail & Related papers  (2026-02-24T23:03:50Z)
• BadCLIP++: Stealthy and Persistent Backdoors in Multimodal Contrastive Learning [73.46118996284888]
  Research on backdoor attacks against multimodal contrastive learning models faces two key challenges: stealthiness and persistence.<n>We propose BadCLIP++, a unified framework that tackles both challenges.<n>For stealthiness, we introduce a semantic-fusion QR micro-trigger that embeds imperceptible patterns near task-relevant regions.<n>For persistence, we stabilize trigger embeddings via radius shrinkage and centroid alignment.
  arXiv  Detail & Related papers  (2026-02-19T08:31:16Z)
• AMGFormer: Adaptive Multi-Granular Transformer for Brain Tumor Segmentation with Missing Modalities [6.461582089537306]
  We propose AMGFormer, achieving significantly improved stability through three synergistic modules.<n>On BraTS 2018, our method achieves 89.33% WT, 82.70% TC, 67.23% ET Dice scores with 0.5% variance across 15 modality combinations.<n>Single-modality ET segmentation shows 40-81% relative improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-01-27T08:29:02Z)
• A Novel Hybrid Deep Learning and Chaotic Dynamics Approach for Thyroid Cancer Classification [3.1331787430863485]
  We present an intelligent classification method that couples an Adaptive Convolutional Neural Network (CNN) with Cohen-Daubechies-Feauveau (CDF9/7) wavelets.<n>We evaluate on the public DDTI thyroid ultrasound dataset (n = 1,638 images; 819 malignant / 819 benign) using 5-fold cross-validation.<n>The proposed method attains 98.17% accuracy, 98.76% sensitivity, 97.58% specificity, 97.55% F1-score, and an AUC of 0.9912.
  arXiv  Detail & Related papers  (2025-09-28T16:46:31Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• AI-driven software for automated quantification of skeletal metastases and treatment response evaluation using Whole-Body Diffusion-Weighted MRI (WB-DWI) in Advanced Prostate Cancer [0.27141618085306923]
  Whole-Body Diffusion-Weighted MRI (WB-DWI) provides two response biomarkers: Total Diffusion Volume (TDV) and global Apparent Diffusion Coefficient (gADC)<n> tracking post-treatment changes of TDV and gADC from manually delineated lesions is cumbersome and increases inter-reader variability.<n>We developed a software to automate this process. Core technologies include: (i) a weakly-supervised Residual U-Net model generating a skeleton probability map to isolate bone; (ii) a statistical framework for WB-DWI intensity normalisation; and (iii) a shallow convolutional neural network that processes outputs
  arXiv  Detail & Related papers  (2025-05-13T22:57:49Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• Patch-Level Contrasting without Patch Correspondence for Accurate and Dense Contrastive Representation Learning [79.43940012723539]
  ADCLR is a self-supervised learning framework for learning accurate and dense vision representation. Our approach achieves new state-of-the-art performance for contrastive methods.
  arXiv  Detail & Related papers  (2023-06-23T07:38:09Z)
• Corneal endothelium assessment in specular microscopy images with Fuchs' dystrophy via deep regression of signed distance maps [48.498376125522114]
  This paper proposes a UNet-based segmentation approach that requires minimal post-processing. It achieves reliable CE morphometric assessment and guttae identification across all degrees of Fuchs' dystrophy.
  arXiv  Detail & Related papers  (2022-10-13T15:34:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.