Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models

• URL: http://arxiv.org/abs/2607.13860v1
• Date: Wed, 15 Jul 2026 14:05:58 GMT
• Title: Towards Enhancing 3D Spatial Reasoning in Medical Multimodal Large Language Models
• Abstract summary: We introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm.<n>Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process.<n>This strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training.
• Score: 8.949206640442354
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in 2D medical image understanding, their extension to 3D volumetric imaging remains hindered by prohibitive annotation costs and dataset opacity. Current data formats, predominantly consisting of rigid Visual Question Answering (VQA) pairs or unstructured final clinical reports, typically fail to capture explicit clinical reasoning. To address this limitation, we introduce a large-scale structured reasoning dataset constructed via a novel slice-wise data synthesis paradigm. Inspired by the genuine diagnostic workflow of radiologists, this paradigm models visual cognition by decomposing the complex 3D reading process, translating global clinical priors into fine-grained, per-slice observations that are subsequently synthesized into an interpretable Chain-of-Thought (CoT). Crucially, this synthesized reasoning framework enforces essential clinical principles: sequential spatial tracking, multi-slice spatial awareness for artifact mitigation, and differential exclusion. To validate this approach, we instruction-tune a standard 2D-pretrained MLLM baseline using the synthesized data to enhance its volumetric comprehension. Comprehensive evaluations across multiple 3D medical benchmarks demonstrate that our method yields significant performance improvements over the 2D baseline. Furthermore, the resulting model exhibits robust spatial reasoning capabilities and rivals resource-intensive native 3D architectures, effectively bridging the performance gap. Ultimately, this data-centric strategy unlocks deep volumetric understanding and highly interpretable clinical logic without requiring computationally expensive 3D-specific pre-training. The complete repository, including datasets and training workflows, is publicly available at https://github.com/2020420145009/hounsfield.

Related papers

• Volumetric Radiology AI in the Era of Multimodal Large Language Models [62.95443321443533]
  Volumetric radiology presents a fundamental representational mismatch.<n>Current large language models (MLLMs) are conditioned on selected two-dimensional (2D) images, compressed visual representations, or report-derived text.<n>Reliable radiology AI requires representations that preserve task-relevant 3D information and systems that can access, verify, and integrate this information.
  arXiv  Detail & Related papers  (2026-08-20T20:23:41Z)
• MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models [59.180043227905294]
  We propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework for efficient 3D medical image understanding.<n>We show that MedPruner enables models such as MedGemma to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens.
  arXiv  Detail & Related papers  (2026-03-12T07:37:00Z)
• Optimizing 3D Diffusion Models for Medical Imaging via Multi-Scale Reward Learning [7.493047521402676]
  This paper presents a method to enhance 3D diffusion models using Reinforcement Learning (RL) with multi-scale feedback.<n>We fine-tune the model using Proximal Policy Optimization (PPO), guided by a novel reward system that integrates both 2D slice-wise assessments and 3D analysis.<n>Our results indicate that incorporating RL feedback effectively steers the generation process toward higher quality distributions.
  arXiv  Detail & Related papers  (2026-03-06T11:30:55Z)
• Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification [69.87877580725768]
  Multimodal Visual Surrogate Compression (MVSC) learns to compress and adapt large 3D sMRI volumes into compact 2D features.<n>MVSC has two key components: a Volume Context that captures global cross-slice context under textual guidance, and an Adaptive Slice Fusion module that aggregates slice-level information in a text-enhanced, patch-wise manner.
  arXiv  Detail & Related papers  (2026-01-29T13:05:46Z)
• Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding [80.66591664266744]
  Lemon is a unified transformer architecture that processes 3D point cloud patches and language tokens as a single sequence.<n>To handle the complexity of 3D data, we develop a structured patchification and tokenization scheme that preserves spatial context.<n>Lemon establishes new state-of-the-art performance across comprehensive 3D understanding and reasoning tasks.
  arXiv  Detail & Related papers  (2025-12-14T20:02:43Z)
• Spatial-ORMLLM: Improve Spatial Relation Understanding in the Operating Room with Multimodal Large Language Model [1.8302608976873713]
  Spatial-ORMLLM is a vision-language model for 3D spatial reasoning in operating rooms.<n>It incorporates 2D modality inputs with rich 3D spatial knowledge extracted by the estimation algorithm.<n>It delivers robust 3D scene reasoning without any additional expert annotations or sensor inputs.
  arXiv  Detail & Related papers  (2025-08-11T17:17:20Z)
• Text-to-CT Generation via 3D Latent Diffusion Model with Contrastive Vision-Language Pretraining [1.447808799346751]
  We introduce a novel architecture for Text-to-CT generation that combines a latent diffusion model with a 3D contrastive vision-language pretraining scheme.<n>Our method offers a scalable and controllable solution for synthesizing clinically meaningful CT volumes from text.
  arXiv  Detail & Related papers  (2025-05-31T16:41:55Z)
• GEAL: Generalizable 3D Affordance Learning with Cross-Modal Consistency [50.11520458252128]
  Existing 3D affordance learning methods struggle with generalization and robustness due to limited annotated data.<n>We propose GEAL, a novel framework designed to enhance the generalization and robustness of 3D affordance learning by leveraging large-scale pre-trained 2D models.<n>GEAL consistently outperforms existing methods across seen and novel object categories, as well as corrupted data.
  arXiv  Detail & Related papers  (2024-12-12T17:59:03Z)
• MG-3D: Multi-Grained Knowledge-Enhanced 3D Medical Vision-Language Pre-training [7.968487067774351]
  3D medical image analysis is pivotal in numerous clinical applications.<n>Large-scale vision-language pre-training remains underexplored in 3D medical image analysis.<n>We propose MG-3D, pre-trained on large-scale data (47.1K)
  arXiv  Detail & Related papers  (2024-12-08T09:45:59Z)
• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Syn3DWound: A Synthetic Dataset for 3D Wound Bed Analysis [28.960666848416274]
  This paper introduces Syn3DWound, an open-source dataset of high-fidelity simulated wounds with 2D and 3D annotations. We propose a benchmarking framework for automated 3D morphometry analysis and 2D/3D wound segmentation.
  arXiv  Detail & Related papers  (2023-11-27T13:59:53Z)
• Interpretable 2D Vision Models for 3D Medical Images [47.75089895500738]
  This study proposes a simple approach of adapting 2D networks with an intermediate feature representation for processing 3D images. We show on all 3D MedMNIST datasets as benchmark and two real-world datasets consisting of several hundred high-resolution CT or MRI scans that our approach performs on par with existing methods.
  arXiv  Detail & Related papers  (2023-07-13T08:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.