MedQA-MM: Shortcuts Behind Medical Visual Reasoning

• URL: http://arxiv.org/abs/2609.03261v2
• Date: Tue, 08 Sep 2026 16:34:49 GMT
• Title: MedQA-MM: Shortcuts Behind Medical Visual Reasoning
• Abstract summary: A benchmark score credits final answers, but not the route by which an item can be answered.<n>In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding.<n>Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition.
• Score: 45.333881843709804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A benchmark score credits final answers, but not the route by which an item can be answered. In medical multimodal multiple-choice questions (MCQs), this distinction matters because a correct answer can be supported by the intended image finding or by benchmark-preserved cues in the wording of answers, non-visual clinical text, visible image text, artificial annotations, or device/context artifacts. We call the resulting score-level overinterpretation reasoning inflation. Here, a route is an observable input path that can support answer selection, not a claim about the model's hidden cognition. Across six medical multimodal MCQ datasets, we separate candidate cues from behavioral evidence through prompt- and image-side audits, modality ablations, and matched repairs that preserve the medical target and answer key. In a 13-configuration open-model panel, full-input accuracy is 62.63%, while text-only and options-only settings achieve 53.96% and 29.71%, respectively. Removing length-gap, absolute/conspicuous, and spatial/prepositional cues lowers accuracy by 6.58, 3.50, and 4.77 percentage points. We also construct MedQA-MM, a 1,000-item shortcut-mitigated subset, where text-only and options-only accuracy fall to 5.21% and 12.33%. This does not imply that models never use images; it shows that medical image-reasoning claims require route-level evidence.

Related papers

• Candidate-Expanding Routing with Permutation-Stabilized Experts for Mixed-Format Medical VQA [2.313903538324245]
  Mixed-format medical visual question answering (VQA) requires stable option selection and machine-readable free-text output.<n>We address both challenges with an answer-text memory, a permutation--stabilized expert, and a sparse candidate- expanding router.
  arXiv  Detail & Related papers  (2026-09-01T09:18:19Z)
• When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning [73.56343820592399]
  We introduce Visualized Task Semantics (VTS), a controlled intervention that moves the question into the image while keeping the source problem and answer fixed.<n>Across six MLLMs and four benchmarks, accuracy drops in all 24 model-task pairs, by 17.8 points on average.<n>We present prompt-region grounding, whose core design aligns the question region with typed semantics and recovers its clean representation from a masked view.
  arXiv  Detail & Related papers  (2026-08-05T11:46:15Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• NeuroQA: A Large-Scale Image-Grounded Benchmark for 3D Brain MRI Understanding [6.217658756255346]
  NeuroQA is a large-scale benchmark for visual question answering in 3D brain magnetic resonance imaging (MRI)<n>It spans ages 5-104 and five clinical domains: Alzheimer's, Parkinson's, tumors, white matter disease, and neurodevelopment.<n>NeuroQA evaluates 11 clinically grounded reasoning skills across Yes/No, multiple-choice, and open-ended formats.
  arXiv  Detail & Related papers  (2026-05-19T21:54:12Z)
• When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models [2.064612766965483]
  We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions)<n>Our experiments reveal several concerning findings.
  arXiv  Detail & Related papers  (2026-03-26T23:04:20Z)
• PSF-Med: Measuring and Explaining Paraphrase Sensitivity in Medical Vision Language Models [2.064612766965483]
  We introduce Paraphrase Sensitivity Failure (PSF)-Med, a benchmark of 19,748 chest Xray questions paired with about 92,000 paraphrases.<n>We measure yes/no flips for the same image and find flip rates from 8% to 58%.<n>We identify a sparse feature at layer 17 that correlates with prompt framing and predicts decision margin shifts.
  arXiv  Detail & Related papers  (2026-02-24T23:03:50Z)
• Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations [19.488236277427358]
  Vision-language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process.<n>We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications.
  arXiv  Detail & Related papers  (2025-10-13T09:28:22Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions [103.20281438405111]
  Multiple-choice question answering (MCQA) is a key competence of performant transformer language models.<n>We employ vocabulary projection and activation patching methods to localize key hidden states that encode relevant information for predicting the correct answer.<n>We show that subsequent layers increase the probability of the predicted answer symbol in vocabulary space, and that this probability increase is associated with a sparse set of attention heads with unique roles.
  arXiv  Detail & Related papers  (2024-07-21T00:10:23Z)
• Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectly [100.60560477391732]
  We promote a problem formulation for reliable visual question answering (VQA) We analyze both their coverage, the portion of questions answered, and risk, the error on that portion. We find that although the best performing models achieve over 71% accuracy on the VQA v2 dataset, introducing the option to abstain limits them to answering less than 8% of the questions to achieve a low risk of error (i.e., 1%) This motivates us to utilize a multimodal selection function to directly estimate the correctness of the predicted answers, which we show can triple the coverage from, for example, 5.0% to 16.7% at
  arXiv  Detail & Related papers  (2022-04-28T16:51:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.