ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding

• URL: http://arxiv.org/abs/2603.19517v1
• Date: Thu, 19 Mar 2026 22:54:28 GMT
• Title: ReXInTheWild: A Unified Benchmark for Medical Photograph Understanding
• Abstract summary: We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics.<n> Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%.<n>A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures.
• Score: 33.868464991344354
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Everyday photographs taken with ordinary cameras are already widely used in telemedicine and other online health conversations, yet no comprehensive benchmark evaluates whether vision-language models can interpret their medical content. Analyzing these images requires both fine-grained natural image understanding and domain-specific medical reasoning, a combination that challenges both general-purpose and specialized models. We introduce ReXInTheWild, a benchmark of 955 clinician-verified multiple-choice questions spanning seven clinical topics across 484 photographs sourced from the biomedical literature. When evaluated on ReXInTheWild, leading multimodal large language models show substantial performance variation: Gemini-3 achieves 78% accuracy, followed by Claude Opus 4.5 (72%) and GPT-5 (68%), while the medical specialist model MedGemma reaches only 37%. A systematic error analysis also reveals four categories of common errors, ranging from low-level geometric errors to high-level reasoning failures and requiring different mitigation strategies. ReXInTheWild provides a challenging, clinically grounded benchmark at the intersection of natural image understanding and medical reasoning. The dataset is available on HuggingFace.

Related papers

• IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations [5.460897030541662]
  We introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles.<n>Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis.<n>Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions.
  arXiv  Detail & Related papers  (2026-06-26T19:18:16Z)
• A Vision-language Framework for Comparative Reasoning in Radiology [68.52471827773482]
  We formulate radiological comparison as an entity-aware cross-image reasoning problem.<n>We construct a large-scale comparative imaging resource derived from routine image-report pairs.<n>We develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases.
  arXiv  Detail & Related papers  (2026-06-04T17:12:47Z)
• JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation [53.571125769505194]
  We introduce JMed48k, a Japanese healthcare licensing benchmark for evaluating vision-language models.<n>Built from official PDF materials released by the Japanese Ministry of Health, Labour and Welfare, JMed48k contains 48,862 exam questions and 20,142 images.<n>We derive JMed48k-Eval, a recent five-year evaluation subset with 12,484 scored questions, including 9,905 text-only questions and 2,579 questions with images.
  arXiv  Detail & Related papers  (2026-05-21T07:20:38Z)
• Medical thinking with multiple images [44.04557445622649]
  We introduce MedThinkVQA, an expert-annotated benchmark for thinking with multiple images.<n>The dataset contains 8,067 cases, including 720 test cases, with an average of 6.62 images per case.<n>On the test set, the best closed-source models, Claude-4.6-Opus, Gemini-3-Pro, and GPT-5.2-xhigh, reach only 57.2%, 55.3%, and 54.9% accuracy.
  arXiv  Detail & Related papers  (2026-04-14T18:51:07Z)
• DeepGI: Explainable Deep Learning for Gastrointestinal Image Classification [0.0]
  The study confronts common endoscopic challenges such as variable lighting, fluctuating camera angles, and frequent imaging artifacts.<n>The best performing models, VGG16 and MobileNetV2, each achieved a test accuracy of 96.5%.<n>The approach includes explainable AI via Grad-CAM visualization, enabling identification of image regions most influential to model predictions.
  arXiv  Detail & Related papers  (2025-11-26T22:35:57Z)
• TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models [54.48710348910535]
  Existing medical reasoning benchmarks primarily focus on analyzing a patient's condition based on an image from a single visit.<n>We introduce TemMed-Bench, the first benchmark designed for analyzing changes in patients' conditions between different clinical visits.
  arXiv  Detail & Related papers  (2025-09-29T17:51:26Z)
• MedBLINK: Probing Basic Perception in Multimodal Language Models for Medicine [12.333678882957377]
  We introduce Medblink, a benchmark designed to probe these models for such perceptual abilities.<n>Medblink spans eight clinically meaningful tasks across multiple imaging modalities and anatomical regions, totaling 1,429 multiple-choice questions over 1,605 images.<n>While human annotators achieve 96.4% accuracy, the best-performing model reaches only 65%.
  arXiv  Detail & Related papers  (2025-08-04T23:19:18Z)
• Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach [57.86418347491272]
  We propose a comprehensive hierarchical classification system, with 404 representative abnormal findings across all body regions.<n>We contribute a dataset containing over 14.5K CT images from multiple planes and all human body regions, and meticulously provide grounding annotations for over 19K abnormalities.<n>We propose OminiAbnorm-CT, which can automatically ground and describe abnormal findings on multi-plane and whole-body CT images based on text queries.
  arXiv  Detail & Related papers  (2025-06-03T17:57:34Z)
• MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning [24.9872402922819]
  Existing medical VQA benchmarks mostly focus on single-image analysis.<n>We introduce MedFrameQA -- the first benchmark that explicitly evaluates multi-image reasoning in medical VQA.
  arXiv  Detail & Related papers  (2025-05-22T17:46:11Z)
• MedIAnomaly: A comparative study of anomaly detection in medical images [26.319602363581442]
  Anomaly detection (AD) aims at detecting abnormal samples that deviate from the expected normal patterns.<n>Despite the emergence of numerous methods for medical AD, the lack of a fair and comprehensive evaluation causes ambiguous conclusions.<n>This paper builds a benchmark with unified comparison to address this problem.
  arXiv  Detail & Related papers  (2024-04-06T06:18:11Z)
• QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge [93.61262892578067]
  Uncertainty in medical image segmentation tasks, especially inter-rater variability, presents a significant challenge. This variability directly impacts the development and evaluation of automated segmentation algorithms. We report the set-up and summarize the benchmark results of the Quantification of Uncertainties in Biomedical Image Quantification Challenge (QUBIQ)
  arXiv  Detail & Related papers  (2024-03-19T17:57:24Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imaging [41.202147558260336]
  We propose a framework that employs saliency maps to obtain soft spatial attention masks that modulate the image features at different scales. ACAT increases the baseline classification accuracy of lesions in brain CT scans from 71.39% to 72.55% and of COVID-19 related findings in lung CT scans from 67.71% to 70.84%.
  arXiv  Detail & Related papers  (2023-03-27T17:43:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.