Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models

• URL: http://arxiv.org/abs/2605.08787v1
• Date: Sat, 09 May 2026 08:16:00 GMT
• Title: Lost in Volume: The CT-SpatialVQA Benchmark for Evaluating Semantic-Spatial Understanding of 3D Medical Vision-Language Models
• Abstract summary: Recent advances in 3D medical vision-language models have enabled joint reasoning over volumetric images and text.<n>It remains unclear whether these models learn spatially grounded anatomy from 3D volumes or rely primarily on learned priors and language correlations.<n>We introduce CT-SpatialVQA, a benchmark designed to evaluate semantic-spatial reasoning in 3D CT data.
• Score: 3.6659638651482536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in 3D medical vision-language models have enabled joint reasoning over volumetric images and text, showing strong performance in medical visual question-answering (VQA) and report generation. Despite this progress, it remains unclear whether these models learn spatially grounded anatomy from 3D volumes or rely primarily on learned priors and language correlations. This uncertainty stems from the lack of systematic evaluation of semantic-spatial reasoning in volumetric medical VLMs for clinically reliable decision support. To address this gap, we introduce CT-SpatialVQA, a benchmark designed to evaluate semantic-spatial reasoning in 3D CT data. The benchmark comprises 9077 clinically grounded question-answer (QA) pairs derived directly from 1601 radiology reports and CT volumes, which are validated via a robust LLM-assisted pipeline with a 95% human consensus agreement rate. Our dataset requires explicit anatomical localization, laterality awareness, structural comparison, and 3D inter-structure relational reasoning. We also introduce a standardized evaluation protocol and benchmark eight 3D medical VLMs, finding severe degradation on semantic-spatial reasoning tasks, averaging 34% accuracy and often below random, highlighting the need for deeper integration of volumetric evidence for trustworthy clinical use.

Related papers

• A Modular Agent for Reliable and Auditable Spatial Relation Verification in CT Scans [7.8047270656196766]
  Modern vision-language models (VLMs) show promising performance on many medical imaging tasks.<n>We present a modular medical imaging agent for binary spatial relation verification in axial CT slices.<n>The best-performing hybrid configuration reaches 94.1% accuracy and 94.2% F1, outperforming direct Qwen2-VL prompting by 42.5 percentage points in accuracy.
  arXiv  Detail & Related papers  (2026-08-21T14:16:46Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• Towards Clinically Interpretable Ophthalmic VQA via Spatially-Grounded Lesion Evidence [51.57735743553791]
  We introduce FundusGround, a new benchmark for clinically interpretable ophthalmic VQA with spatially-grounded lesion evidence.<n>Built upon this structured lesion evidence, 72,706 questions are then generated spanning four formats.<n>Experiments demonstrate that incorporating lesion-level visual evidence consistently improves model performance and transparency.
  arXiv  Detail & Related papers  (2026-05-21T12:37:03Z)
• MedScribe: Clinically Grounded CT Reporting through Agentic Workflows [13.40306812882295]
  Vision-language models (VLMs) have shown potential for automated radiology report generation.<n>We introduce MedScribe, a hypothesis-driven framework that reformulates report generation as an iterative evidence acquisition process.
  arXiv  Detail & Related papers  (2026-05-03T08:32:40Z)
• EXACT: an explainable anomaly-aware vision foundation model for analysis of 3D chest CT [29.0378459959757]
  EXACT is an anomaly-aware foundation model for three-dimensional chest CT.<n>It learns spatially resolved representations from paired clinical scans and radiology reports.<n> EXACT shows consistent improvements across clinically relevant CT tasks.
  arXiv  Detail & Related papers  (2026-04-27T07:57:47Z)
• LoV3D: Grounding Cognitive Prognosis Reasoning in Longitudinal 3D Brain MRI via Regional Volume Assessments [14.481985722970238]
  We present LoV3D, a pipeline for training 3D vision-language models, which reads longitudinal T1-weighted brain MRI.<n>The pipeline grounds the final diagnosis by enforcing label consistency, longitudinal coherence, and biological plausibility.<n>On a subject-level held-out ADNI test set, LoV3D achieves 93.7% three-class diagnostic accuracy.
  arXiv  Detail & Related papers  (2026-03-12T15:40:59Z)
• Interpretable Aneurysm Classification via 3D Concept Bottleneck Models: Integrating Morphological and Hemodynamic Clinical Features [0.6445605125467574]
  We propose an end-to-end 3D Concept Bottleneck framework that maps high-dimensional neuroimaging features to a discrete set of morphological and hemodynamic concepts for aneurysm identification.<n>Our results demonstrate a peak task classification accuracy of 93.33% +/- 4.5% for the ResNet-34 architecture and 91.43% +/- 5.8% for the DenseNet-121 model.
  arXiv  Detail & Related papers  (2026-03-08T01:03:48Z)
• OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis [53.01523944168442]
  Clinical interpretation relies on both slice-driven local features and volume-driven spatial representations.<n>Existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding.<n>We present OmniCT, a powerful unified slice-volume LVLM for CT scenarios.
  arXiv  Detail & Related papers  (2026-02-18T00:42:41Z)
• Med3D-R1: Incentivizing Clinical Reasoning in 3D Medical Vision-Language Models for Abnormality Diagnosis [20.302134776419955]
  We propose a reinforcement learning framework with a two-stage training process: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)<n>In RL stage, we redesign the consistency reward to explicitly promote coherent, step-by-step diagnostic reasoning.<n>Our model attains state-of-the-art accuracies of 41.92% on CT-RATE and 44.99% on RAD-ChestCT.
  arXiv  Detail & Related papers  (2026-02-01T12:43:11Z)
• S-Chain: Structured Visual Chain-of-Thought For Medicine [81.97605645734741]
  We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT)<n>The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability.<n>S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical vision-language models.
  arXiv  Detail & Related papers  (2025-10-26T15:57:14Z)
• RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis [56.57177181778517]
  RadGenome-Chest CT is a large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. We leverage the latest powerful universal segmentation and large language models to extend the original datasets.
  arXiv  Detail & Related papers  (2024-04-25T17:11:37Z)
• CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios [53.94122089629544]
  We introduce CT-GLIP (Grounded Language-Image Pretraining with CT scans), a novel method that constructs organ-level image-text pairs to enhance multimodal contrastive learning. Our method, trained on a multimodal CT dataset comprising 44,011 organ-level vision-text pairs from 17,702 patients across 104 organs, demonstrates it can identify organs and abnormalities in a zero-shot manner using natural languages.
  arXiv  Detail & Related papers  (2024-04-23T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.