Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence

• URL: http://arxiv.org/abs/2608.12928v1
• Date: Thu, 13 Aug 2026 08:07:22 GMT
• Title: Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
• Abstract summary: The benchmark comprises image-containing questions spanning diverse medical specialties and visual domains.<n>We evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models.
• Score: 2.779050345468656
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a Polish-language medical visual question answering (VQA) benchmark, built from Polish Board Certification Examination questions for licensed physicians and dentists pursuing specialist certification. The benchmark comprises image-containing questions spanning diverse medical specialties and visual domains, together with a text-only question answering (QA) control set. We evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models. The task remains challenging: the best model achieves 79.0\% accuracy on the full VQA set, and only GPT-5.6 surpasses the approximate human reference on the subset with available candidate responses; all other evaluated models perform worse than humans. To assess visual grounding, we compare complete inputs with configurations omitting the image, the question, or both, and categorize questions by image importance. Models derive more useful information from the question text than from the image and perform worse on image-dominant questions. Across both QA and VQA, they nevertheless achieve above-chance accuracy from the answer choices alone, showing that non-trivial performance can persist even when key task components are missing.

Related papers

• WearVQA: A Visual Question Answering Benchmark for Wearables in Egocentric Authentic Real-world scenarios [19.156760664417718]
  We introduce WearVQA, the first benchmark specifically designed to evaluate the Visual Question Answering capabilities of multi-model AI assistant on wearable devices like smart glasses.<n>WearVQA reflects the unique challenges of ego-centric interaction-where visual inputs may be occluded, poorly lit, unzoomed, or blurry.<n>The benchmark comprises 2,520 carefully curated image-question-answer triplets, spanning 7 diverse image domains.
  arXiv  Detail & Related papers  (2025-11-27T06:44:49Z)
• CommVQA: Situating Visual Question Answering in Communicative Contexts [16.180130883242672]
  We introduce CommVQA, a dataset consisting of images, image descriptions, real-world communicative scenarios where the image might appear. We show that access to contextual information is essential for solving CommVQA, leading to the highest performing VQA model.
  arXiv  Detail & Related papers  (2024-02-22T22:31:39Z)
• Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts [54.072432123447854]
  Visual question answering (VQA) is the task of answering questions about an image. Answering the question requires commonsense knowledge, world knowledge, and reasoning about ideas and concepts not present in the image. We propose a framework that uses language guidance (LG) in the form of rationales, image captions, scene graphs, etc to answer questions more accurately.
  arXiv  Detail & Related papers  (2023-10-31T03:54:11Z)
• Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation [64.64849950642619]
  We develop an evaluation framework inspired by formal semantics for evaluating text-to-image models. We show that Davidsonian Scene Graph (DSG) produces atomic and unique questions organized in dependency graphs. We also present DSG-1k, an open-sourced evaluation benchmark that includes 1,060 prompts.
  arXiv  Detail & Related papers  (2023-10-27T16:20:10Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Localized Questions in Medical Visual Question Answering [2.005299372367689]
  Visual Question Answering (VQA) models aim to answer natural language questions about given images. Existing medical VQA models typically focus on answering questions that refer to an entire image. This paper proposes a novel approach for medical VQA that addresses this limitation by developing a model that can answer questions about image regions.
  arXiv  Detail & Related papers  (2023-07-03T14:47:18Z)
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering [56.25766322554655]
  Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery. We propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef 2019.
  arXiv  Detail & Related papers  (2023-05-17T17:50:16Z)
• BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models [47.64219291655723]
  We introduce a new test set for visual question answering (VQA) called BinaryVQA to push the limits of VQA models. Our dataset includes 7,800 questions across 1,024 images and covers a wide variety of objects, topics, and concepts. Around 63% of the questions have positive answers.
  arXiv  Detail & Related papers  (2023-01-28T00:03:44Z)
• Self-supervised vision-language pretraining for Medical visual question answering [9.073820229958054]
  We propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining. The proposed method achieves state-of-the-art performance on all the three public medical VQA datasets.
  arXiv  Detail & Related papers  (2022-11-24T13:31:56Z)
• A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge [39.788346536244504]
  A-OKVQA is a crowdsourced dataset composed of about 25K questions. We demonstrate the potential of this new dataset through a detailed analysis of its contents.
  arXiv  Detail & Related papers  (2022-06-03T17:52:27Z)
• Human-Adversarial Visual Question Answering [62.30715496829321]
  We benchmark state-of-the-art VQA models against human-adversarial examples. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples.
  arXiv  Detail & Related papers  (2021-06-04T06:25:32Z)
• SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions [66.86887670416193]
  We show that state-of-the-art VQA models have comparable performance in answering perception and reasoning questions, but suffer from consistency problems. To address this shortcoming, we propose an approach called Sub-Question-aware Network Tuning (SQuINT) We show that SQuINT improves model consistency by 5%, also marginally improving performance on the Reasoning questions in VQA, while also displaying better attention maps.
  arXiv  Detail & Related papers  (2020-01-20T01:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.