CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs

• URL: http://arxiv.org/abs/2606.08420v1
• Date: Sun, 07 Jun 2026 02:41:05 GMT
• Title: CheXanatomy: Anatomy-Aware Vision-Language Modeling for Chest Radiographs
• Abstract summary: CheXanatomy is a framework that integrates explicit anatomical knowledge into a pretrained vision-language model.<n>To enable scalable supervision, we synthesize realistic chest radiographs from CT volumes and forward-project CT segmentation labels.<n>We evaluate the approach on synthetic and real chest radiographs against a U-Net baseline, including ablations on model scale, input resolution, and vision encoder fine-tuning.
• Score: 3.369056928100267
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) pretrained on large-scale image-text pairs demonstrate strong image-level understanding, but are primarily optimized for global alignment and do not explicitly encode fine-grained anatomical structure, limiting their suitability for spatially precise tasks such as segmentation. We introduce CheXanatomy, a framework that integrates explicit anatomical knowledge into a pretrained VLM through autoregressive token-space supervision. Instead of adding task-specific decoder heads, the model is trained to generate anatomical segmentation masks via next-token prediction. To enable scalable supervision, we synthesize realistic chest radiographs from CT volumes and forward-project CT segmentation labels to obtain anatomically consistent 2D masks. We evaluate the approach on synthetic and real chest radiographs against a U-Net baseline, including ablations on model scale, input resolution, and vision encoder fine-tuning. Autoregressive anatomical supervision achieves performance comparable to specialized convolutional models in-distribution and demonstrates improved geometric robustness under domain shift to real CXR data. In addition, anatomy-pretrained models exhibit improved sample efficiency when adapting to novel localization tasks under limited supervision. Larger models and higher input image resolution improve performance, while vision encoder fine-tuning has limited effect. These results show that embedding anatomical structure directly into the generative objective promotes spatially grounded representations and supports anatomy-aware medical vision-language modeling.

Related papers

• Towards Voxel Spacing Consistency for Medical Image Segmentation [56.992050146446495]
  We study the correlation between image resampling and segmentation, and propose Consispace, a semantic-aware resampling framework.<n> Consispace achieves consistent voxel spacing in the axial direction while preserving anatomical and semantic consistency.
  arXiv  Detail & Related papers  (2026-06-30T15:42:23Z)
• SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis [51.81845420301345]
  SonoCLIP is a vision-language foundation model for fetal ultrasound analysis.<n>It integrates segmentation masks as mask-channel visual prompts within the vision encoder.<n>It achieves superior zero-shot transfer performance under both global and mask-guided inference.
  arXiv  Detail & Related papers  (2026-06-28T20:04:49Z)
• Anatomy-Anchored Self-Supervision: Distilling Vision Foundation Models for Invariant Ultrasound Representation [29.61647712180762]
  Self-supervised pre-training paradigm has gained increasing prominence for learning transferable representations in medical imaging.<n>We propose an anatomy-anchored ultrasound self-supervision framework ANAUS that shifts representation learning from generic visual regions to clinically meaningful anatomical structures.
  arXiv  Detail & Related papers  (2026-05-25T03:52:58Z)
• LGESynthNet: Controlled Scar Synthesis for Improved Scar Segmentation in Cardiac LGE-MRI Imaging [21.180056139848528]
  We introduce LGE SynthNet, a latent diffusion-based framework for controllable enhancement synthesis.<n>The model integrates: (a) a reward model for conditioning-specific supervision, (b) a captioning module for descriptive text prompts, and (c) a biomedical text encoder.<n> trained on just 429 images (79 patients), it produces anatomically coherent samples.
  arXiv  Detail & Related papers  (2026-03-18T23:40:33Z)
• X-GRM: Large Gaussian Reconstruction Model for Sparse-view X-rays to Computed Tomography [89.84588038174721]
  Computed Tomography serves as an indispensable tool in clinical, providing non-invasive visualization of internal anatomical structures.<n>Existing CT reconstruction works are limited to small-capacity model architecture and inflexible volume representation.<n>We present X-GRM, a large feedforward model for reconstructing 3D CT volumes from sparse-view 2D X-ray projections.
  arXiv  Detail & Related papers  (2025-05-21T08:14:10Z)
• PathSegDiff: Pathology Segmentation using Diffusion model representations [63.20694440934692]
  We propose PathSegDiff, a novel approach for histopathology image segmentation that leverages Latent Diffusion Models (LDMs) as pre-trained featured extractors.<n>Our method utilizes a pathology-specific LDM, guided by a self-supervised encoder, to extract rich semantic information from H&E stained histopathology images.<n>Our experiments demonstrate significant improvements over traditional methods on the BCSS and GlaS datasets.
  arXiv  Detail & Related papers  (2025-04-09T14:58:21Z)
• DAMamba: Vision State Space Model with Dynamic Adaptive Scan [51.81060691414399]
  State space models (SSMs) have recently garnered significant attention in computer vision.<n>We propose Dynamic Adaptive Scan (DAS), a data-driven method that adaptively allocates scanning orders and regions.<n>Based on DAS, we propose the vision backbone DAMamba, which significantly outperforms current state-of-the-art vision Mamba models in vision tasks.
  arXiv  Detail & Related papers  (2025-02-18T08:12:47Z)
• MASSM: An End-to-End Deep Learning Framework for Multi-Anatomy Statistical Shape Modeling Directly From Images [1.9029890402585894]
  We introduce MASSM, a novel end-to-end deep learning framework that simultaneously localizes multiple anatomies, estimates population-level statistical representations, and delineates shape representations directly in image space. Our results show that MASSM, which delineates anatomy in image space and handles multiple anatomies through a multitask network, provides superior shape information compared to segmentation networks for medical imaging tasks.
  arXiv  Detail & Related papers  (2024-03-16T20:16:37Z)
• Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models [11.835841459200632]
  We propose a diffusion model-based method that supports anatomically-controllable medical image generation. We additionally introduce a random mask ablation training algorithm to enable conditioning on a selected combination of anatomical constraints. SegGuidedDiff reaches a new state-of-the-art in the faithfulness of generated images to input anatomical masks.
  arXiv  Detail & Related papers  (2024-02-07T19:35:09Z)
• Attentive Symmetric Autoencoder for Brain MRI Segmentation [56.02577247523737]
  We propose a novel Attentive Symmetric Auto-encoder based on Vision Transformer (ViT) for 3D brain MRI segmentation tasks. In the pre-training stage, the proposed auto-encoder pays more attention to reconstruct the informative patches according to the gradient metrics. Experimental results show that our proposed attentive symmetric auto-encoder outperforms the state-of-the-art self-supervised learning methods and medical image segmentation models.
  arXiv  Detail & Related papers  (2022-09-19T09:43:19Z)
• Improving anatomical plausibility in medical image segmentation via hybrid graph neural networks: applications to chest x-ray analysis [3.3382651833270587]
  We introduce HybridGNet, an encoder-decoder neural architecture that leverages standard convolutions for image feature encoding and graph convolutional neural networks (GCNNs) to decode plausible representations of anatomical structures. A novel image-to-graph skip connection layer allows localized features to flow from standard convolutional blocks to GCNN blocks, and show that it improves segmentation accuracy.
  arXiv  Detail & Related papers  (2022-03-21T13:37:23Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Learning to Segment Anatomical Structures Accurately from One Exemplar [34.287877547953194]
  Methods that permit to produce accurate anatomical structure segmentation without using a large amount of fully annotated training images are highly desirable. We propose Contour Transformer Network (CTN), a one-shot anatomy segmentor including a naturally built-in human-in-the-loop mechanism. We demonstrate that our one-shot learning method significantly outperforms non-learning-based methods and performs competitively to the state-of-the-art fully supervised deep learning approaches.
  arXiv  Detail & Related papers  (2020-07-06T20:27:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.