MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT

• URL: http://arxiv.org/abs/2609.03690v1
• Date: Thu, 03 Sep 2026 11:27:22 GMT
• Title: MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT
• Abstract summary: We introduce MetaStructAtlas, a large-scale dataset for grounded whole-body PET/CT interpretation.<n>We also develop MetaStructVQA, a standardized 3D grounded visual question-answering benchmark.
• Score: 28.599818943596777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The joint interpretation of metabolic function and anatomical structure is essential for clinical diagnosis in whole-body PET/CT. Although recent advances in 3D medical vision-language models have demonstrated remarkable progress, current efforts are limited to regional CT imaging, leaving a critical void in comprehensive whole-body PET/CT analysis. In this work, we introduce MetaStructAtlas, a large-scale dataset for grounded whole-body PET/CT interpretation that synthesizes multimodal imaging with integrated anatomical, metabolic, and semantic annotations. MetaStructAtlas provides 490 co-registered 3D PET and CT volumes with 50,470 organ-level segmentation masks and grounded radiology reports. To facilitate interactive reasoning, we further developed MetaStructVQA, a standardized 3D grounded visual question-answering benchmark containing 100,565 QA pairs. This framework explicitly links diagnostic queries to visual evidence across modalities, encompassing anatomical, morphological, and metabolic characteristics. Finally, we evaluate state-of-the-art 3D medical VLMs on MetaStructVQA, establishing a robust foundation for multimodal representation learning and integrated whole-body reasoning in nuclear medicine.

Related papers

• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis [53.01523944168442]
  Clinical interpretation relies on both slice-driven local features and volume-driven spatial representations.<n>Existing Large Vision-Language Models (LVLMs) remain fragmented in CT slice versus volumetric understanding.<n>We present OmniCT, a powerful unified slice-volume LVLM for CT scenarios.
  arXiv  Detail & Related papers  (2026-02-18T00:42:41Z)
• Uncovering Modality Discrepancy and Generalization Illusion for General-Purpose 3D Medical Segmentation [26.70378621396797]
  3D medical foundation models are envisioned as versatile tools with offer general-purpose capabilities.<n>This dataset comprises 490 whole-body PET/CT and 464 whole-body PET/MRI scans.
  arXiv  Detail & Related papers  (2026-02-07T17:54:10Z)
• A Generalist Foundation Model for Total-body PET/CT Enables Diagnostic Reporting and System-wide Metabolic Profiling [17.709593152803695]
  SDF-HOLO is a multimodal foundation model for holistic total-body PET/CT, pre-trained on more than 10,000 patients.<n>It decouples CT and PET representation learning with dual-stream encoders and couples them through a cross-modal interaction module.<n>The model enables system-wide metabolic profiling and reveals tumor-associated fingerprints of inter-organ metabolic network interactions.
  arXiv  Detail & Related papers  (2026-01-19T08:30:09Z)
• TGC-Net: A Structure-Aware and Semantically-Aligned Framework for Text-Guided Medical Image Segmentation [56.09179939570486]
  We propose TGC-Net, a CLIP-based framework focusing on parameter-efficient, task-specific adaptations.<n>TGC-Net achieves state-of-the-art performance with substantially fewer trainable parameters, including notable Dice gains on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-12-24T12:06:26Z)
• Vision-Language Models for Automated 3D PET/CT Report Generation [13.781844347232079]
  AutoPET/CT report generation is increasingly important for reducing clinical workload.<n>PETRG-3D is an end-to-end 3D dual-branch framework that encodes PET and CT volumes and incorporates style-adaptive prompts.<n>PETRG-Score is a lymphoma-specific evaluation protocol that measures metabolic and structural findings across curated anatomical regions.
  arXiv  Detail & Related papers  (2025-11-25T10:07:57Z)
• PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting [10.800541081358132]
  We introduce a large-scale dataset comprising over 11,000 lesion-level descriptions paired with 3D segmentations from more than 5,000 PET/CT exams.<n>Building upon this dataset, we propose PETAR-4B, a 3D mask-aware vision-language model that integrates PET, CT, and lesion contours for spatially grounded report generation.
  arXiv  Detail & Related papers  (2025-10-31T17:49:01Z)
• Rethinking Whole-Body CT Image Interpretation: An Abnormality-Centric Approach [57.86418347491272]
  We propose a comprehensive hierarchical classification system, with 404 representative abnormal findings across all body regions.<n>We contribute a dataset containing over 14.5K CT images from multiple planes and all human body regions, and meticulously provide grounding annotations for over 19K abnormalities.<n>We propose OminiAbnorm-CT, which can automatically ground and describe abnormal findings on multi-plane and whole-body CT images based on text queries.
  arXiv  Detail & Related papers  (2025-06-03T17:57:34Z)
• Developing a PET/CT Foundation Model for Cross-Modal Anatomical and Functional Imaging [39.59895695500171]
  We introduce the Cross-Fraternal Twin Masked Autoencoder (FratMAE), a novel framework that effectively integrates whole-body anatomical and functional information.<n>FratMAE captures intricate cross-modal relationships and global uptake patterns, achieving superior performance on downstream tasks.
  arXiv  Detail & Related papers  (2025-03-04T17:49:07Z)
• RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis [56.57177181778517]
  RadGenome-Chest CT is a large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. We leverage the latest powerful universal segmentation and large language models to extend the original datasets.
  arXiv  Detail & Related papers  (2024-04-25T17:11:37Z)
• CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios [53.94122089629544]
  We introduce CT-GLIP (Grounded Language-Image Pretraining with CT scans), a novel method that constructs organ-level image-text pairs to enhance multimodal contrastive learning. Our method, trained on a multimodal CT dataset comprising 44,011 organ-level vision-text pairs from 17,702 patients across 104 organs, demonstrates it can identify organs and abnormalities in a zero-shot manner using natural languages.
  arXiv  Detail & Related papers  (2024-04-23T17:59:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.