Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations

• URL: http://arxiv.org/abs/2608.28092v1
• Date: Fri, 28 Aug 2026 08:59:27 GMT
• Title: Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations
• Abstract summary: We construct SPAR-Bench, eight probes over multi-organ abdominal CT.<n>We apply them to five architectural configurations and three medical foundation models, frozen and finetuned.<n>Our results suggest these encoders carry a map of where organs usually lie, and little of the machinery for comparing structures within a particular patient.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Interpreting a CT scan means comparing structures on either side, judging how far apart organs sit, and knowing where each one belongs. Medical vision encoders are evaluated on diagnostic accuracy, or through assembled multimodal systems where a failure is hard to attribute, so it remains unclear whether their representations support any of this. We construct SPAR-Bench, eight probes over multi-organ abdominal CT that separate coordinate localization, relational reasoning, and spatial queries, and apply them to five architectural configurations and three medical foundation models, frozen and finetuned. Probes that ask for a comparison within the slice stay at chance, and neither pretraining scale, finetuning, nor architecture closes the gap. Probes that appear solved in domain fall to chance under zero-shot transfer, indicating that their accuracy reflects recall of canonical anatomy rather than computation over the image. Reading the same frozen features with a pooled head rather than the full set of tokens moves relational recovery from 0.7% to 67.8%, so pooled probing understates what a representation holds. Questions the encoders answer well are answered at chance by four open-weight MLLMs. Our results suggest these encoders carry a map of where organs usually lie, and little of the machinery for comparing structures within a particular patient. Code and data will be available at https://spar-bench.github.io.

Related papers

• Frozen Brain-MRI Foundation Models Are Site Fingerprints [0.0]
  Frozen foundation-model embeddings are increasingly used as off-the-shelf brain-MRI representations.<n>We audit what they actually encode and find that site is a large, intrinsic component of the representation.<n>We recommend site-audited use of frozen brain-MRI FMs and release an open audit toolkit.
  arXiv  Detail & Related papers  (2026-08-10T23:03:21Z)
• Big, Bright, or Invisible: A Frozen-Feature Benchmark of 3D CT Foundation Models [30.102620002840904]
  3D CT foundation models could assist routine interpretation by providing generalizable representations of anatomy and pathology.<n>We benchmark ten frozen CT encoders across three cohorts of thoracic CT scans.<n>We find no universal state-of-the-art, with rankings fluctuating significantly depending on the evaluation context.
  arXiv  Detail & Related papers  (2026-08-06T12:34:17Z)
• Self-supervision drives representational convergence in medical foundation models more than clinical supervision [2.8356496885865017]
  We present a controlled dissection across 18 image and 7 text encoders, all open-weight and run locally, spanning 7M to 27B parameters and five imaging modalities.<n>We train encoders that vary only the objective under fixed data, architecture, and scale, and reproduce the effect in a synthetic model.<n> Convergence is modest but above a random floor, driven by the self-supervised objective, not clinical supervision.
  arXiv  Detail & Related papers  (2026-07-22T15:25:05Z)
• A Vision-language Framework for Comparative Reasoning in Radiology [68.52471827773482]
  We formulate radiological comparison as an entity-aware cross-image reasoning problem.<n>We construct a large-scale comparative imaging resource derived from routine image-report pairs.<n>We develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases.
  arXiv  Detail & Related papers  (2026-06-04T17:12:47Z)
• SAMe: A Semantic Anatomy Mapping Engine for Robotic Ultrasound [60.64817443833982]
  We present SAMe, a semantic anatomy mapping engine that provides robotic ultrasound with an explicit anatomical prior layer.<n> SAMe addresses scan initiation as a target-to-anatomy-to-action process.<n>It instantiates a patient-specific anatomical representation for the grounded targets from a single external body image.
  arXiv  Detail & Related papers  (2026-04-28T13:44:09Z)
• 3D Medical Image Segmentation based on multi-scale MPU-Net [5.393743755706745]
  This paper proposes a tumor segmentation model MPU-Net for patient volume CT images. It is inspired by Transformer with a global attention mechanism. Compared with the benchmark model U-Net, MPU-Net shows excellent segmentation results.
  arXiv  Detail & Related papers  (2023-07-11T20:46:19Z)
• Med-Query: Steerable Parsing of 9-DoF Medical Anatomies with Query Embedding [14.901279446640393]
  We propose a steerable, robust, and efficient computing framework for detection, identification, and segmentation of anatomies in CT scans.<n>Considering the complicated shapes, sizes, and orientations of anatomies, we present a nine degrees of freedom (9-DoF) pose estimation solution in full 3D space.<n>We have validated our method on three medical imaging parsing tasks: ribs, spine, and abdominal organs.
  arXiv  Detail & Related papers  (2022-12-05T04:04:21Z)
• Attentive Symmetric Autoencoder for Brain MRI Segmentation [56.02577247523737]
  We propose a novel Attentive Symmetric Auto-encoder based on Vision Transformer (ViT) for 3D brain MRI segmentation tasks. In the pre-training stage, the proposed auto-encoder pays more attention to reconstruct the informative patches according to the gradient metrics. Experimental results show that our proposed attentive symmetric auto-encoder outperforms the state-of-the-art self-supervised learning methods and medical image segmentation models.
  arXiv  Detail & Related papers  (2022-09-19T09:43:19Z)
• Weakly-supervised Biomechanically-constrained CT/MRI Registration of the Spine [72.85011943179894]
  We propose a weakly-supervised deep learning framework that preserves the rigidity and the volume of each vertebra while maximizing the accuracy of the registration. We specifically design these losses to depend only on the CT label maps since automatic vertebra segmentation in CT gives more accurate results contrary to MRI. Our results show that adding the anatomy-aware losses increases the plausibility of the inferred transformation while keeping the accuracy untouched.
  arXiv  Detail & Related papers  (2022-05-16T10:59:55Z)
• A unified 3D framework for Organs at Risk Localization and Segmentation for Radiation Therapy Planning [56.52933974838905]
  Current medical workflow requires manual delineation of organs-at-risk (OAR) In this work, we aim to introduce a unified 3D pipeline for OAR localization-segmentation. Our proposed framework fully enables the exploitation of 3D context information inherent in medical imaging.
  arXiv  Detail & Related papers  (2022-03-01T17:08:41Z)
• Chest x-ray automated triage: a semiologic approach designed for clinical implementation, exploiting different types of labels through a combination of four Deep Learning architectures [83.48996461770017]
  This work presents a Deep Learning method based on the late fusion of different convolutional architectures. We built four training datasets combining images from public chest x-ray datasets and our institutional archive. We trained four different Deep Learning architectures and combined their outputs with a late fusion strategy, obtaining a unified tool.
  arXiv  Detail & Related papers  (2020-12-23T14:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.