Medical thinking with multiple images

• URL: http://arxiv.org/abs/2604.16506v1
• Date: Tue, 14 Apr 2026 18:51:07 GMT
• Title: Medical thinking with multiple images
• Abstract summary: We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images.<n>The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case.<n>On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy.
• Score: 44.04557445622649
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models perform well on many medical QA benchmarks, but real clinical reasoning often requires integrating evidence across multiple images rather than interpreting a single view. We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images, where models must interpret each image, combine cross-view evidence, and answer diagnostic questions with intermediate supervision and step-level evaluation. The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case, substantially denser than prior work, whose expert-level benchmarks use at most 1.43 images per case. On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy, while GPT-5-mini and GPT-5-nano reach 39.7% and 30.8%. Strong open-source models lag behind, led by Qwen3.5-397B-A17B at 52.2% and Qwen3.5-27B at 50.6%. Further analysis identifies grounded multi-image reasoning as the main bottleneck: models often fail to extract, align, and compose evidence across views before higher-level inference can help. Providing expert single-image cues and cross-image summaries improves performance, whereas replacing them with self-generated intermediates reduces accuracy. Step-level analysis shows that over 70% of errors arise from image reading and cross-view integration. Scaling results further show that additional inference-time computation helps only when visual grounding is already reliable; when early evidence extraction is weak, longer reasoning yields limited or unstable gains and can amplify misread cues. These results suggest that the key challenge is not reasoning length alone, but reliable mechanisms for grounding, aligning, and composing distributed evidence across real-world multimodal clinical inputs.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• MC-CXR: A Multi-Context Chest X-ray Benchmark for Context-Induced Disruption in Vision-Language Models [2.2090506971647144]
  Vision-language models (VLMs) are increasingly used in clinical pipelines where a chest X-ray is interpreted alongside retrieved reports, preliminary notes, or prior imaging.<n>Existing benchmarks measure whether models answer correctly in isolation, but not whether they preserve a correct image-only decision when plausible context conflicts with the image.<n>We introduce Multi-Context Chest X-ray (MC-CXR), a benchmark of 240 cases expanded into 2,522 instances that isolates context-induced disruption through paired perturbation.
  arXiv  Detail & Related papers  (2026-08-25T06:28:17Z)
• Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty? [59.923111838399144]
  We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
  arXiv  Detail & Related papers  (2026-05-12T19:33:28Z)
• ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding [33.868464991344354]
  We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics.<n> Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%.<n>A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures.
  arXiv  Detail & Related papers  (2026-03-19T22:54:28Z)
• MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents [57.32877731797049]
  MultiHaystack is the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions.<n>We find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus.
  arXiv  Detail & Related papers  (2026-03-05T21:43:02Z)
• ProImage-Bench: Rubric-Based Evaluation for Professional Image Generation [151.75112778479468]
  We study professional image generation, where a model must synthesize scientifically precise illustrations from technical descriptions.<n>For 654 figures collected from real textbooks and technical reports, we construct detailed image instructions and a hierarchy of rubrics that decompose correctness into 6,076 criteria and 44,131 binary checks.<n>We benchmark several representative text-to-image models on ProImage-Bench and find that, despite strong open-domain performance, the best base model reaches only 0.791 accuracy and 0.553 criterion score overall.
  arXiv  Detail & Related papers  (2025-12-13T07:13:43Z)
• MIRAGE: Agentic Framework for Multimodal Misinformation Detection with Web-Grounded Reasoning [0.6475163438744868]
  We present MIRAGE, an inference-time, model-pluggable agentic framework that decomposes multimodal verification into four sequential modules.<n> visual veracity assessment detects AI-generated images, cross-modal consistency analysis identifies out-of-context repurposing, retrieval-augmented factual checking grounds claims in web evidence.<n>MIRAGE orchestrates vision-language model reasoning with targeted web retrieval, outputs structured and citation-linked rationales.
  arXiv  Detail & Related papers  (2025-10-20T14:40:26Z)
• COREVQA: A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark [3.5018278981067685]
  COREVQA (Crowd Observations and Reasoning Entailment) is a benchmark of 5608 image and synthetically generated true/false statement pairs.<n>Our results show that even the top-performing VLMs achieve accuracy below 80%, with other models performing substantially worse.
  arXiv  Detail & Related papers  (2025-07-17T04:47:47Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning [24.9872402922819]
  Existing medical VQA benchmarks mostly focus on single-image analysis.<n>We introduce MedFrameQA -- the first benchmark that explicitly evaluates multi-image reasoning in medical VQA.
  arXiv  Detail & Related papers  (2025-05-22T17:46:11Z)
• ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models [48.99485386990197]
  We conduct a case study using a synthetic dataset solvable only through visual reasoning.<n>We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions.<n>Although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%.
  arXiv  Detail & Related papers  (2025-05-19T17:59:27Z)
• FLIP Reasoning Challenge [20.706469085872516]
  This paper introduces the FLIP dataset, a benchmark for evaluating AI reasoning capabilities based on human verification tasks.<n>FLIP challenges present users with two orderings of 4 images, requiring them to identify the coherent one.<n>Our experiments evaluate state-of-the-art models, leveraging both vision-language models (VLMs) and large language models (LLMs)
  arXiv  Detail & Related papers  (2025-04-16T17:07:16Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.