RadGPT: Constructing 3D Image-Text Tumor Datasets

• URL: http://arxiv.org/abs/2501.04678v2
• Date: Tue, 19 Aug 2025 17:05:14 GMT
• Title: RadGPT: Constructing 3D Image-Text Tumor Datasets
• Authors: Pedro R. A. S. Bassi, Mehmet Can Yavuz, Kang Wang, Xiaoxi Chen, Wenxuan Li, Sergio Decherchi, Andrea Cavalli, Yang Yang, Alan Yuille, Zongwei Zhou,
• Abstract summary: We present AbdomenAtlas 3.0, the first public, high-quality abdominal CT dataset with detailed, expert-reviewed radiology reports.<n>All reports are paired with per-voxel masks and they describe liver, kidney and pancreatic tumors.<n>Our results show that segmentation strongly improves tumor detection in AI-made reports.
• Score: 13.909446077455323
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Cancers identified in CT scans are usually accompanied by detailed radiology reports, but publicly available CT datasets often lack these essential reports. This absence limits their usefulness for developing accurate report generation AI. To address this gap, we present AbdomenAtlas 3.0, the first public, high-quality abdominal CT dataset with detailed, expert-reviewed radiology reports. All reports are paired with per-voxel masks and they describe liver, kidney and pancreatic tumors. AbdomenAtlas 3.0 has 9,262 triplets of CT, mask and report--3,955 with tumors. These CT scans come from 17 public datasets. Besides creating the reports for these datasets, we expanded their number of tumor masks by 4.2x, identifying 3,011 new tumor cases. Notably, the reports in AbdomenAtlas 3.0 are more standardized, and generated faster than traditional human-made reports. They provide details like tumor size, location, attenuation and surgical resectability. These reports were created by 12 board-certified radiologists using our proposed RadGPT, a novel framework that converted radiologist-revised tumor segmentation masks into structured and narrative reports. Besides being a dataset creation tool, RadGPT can also become a fully-automatic, segmentation-assisted report generation method. We benchmarked this method and 5 state-of-the-art report generation vision-language models. Our results show that segmentation strongly improves tumor detection in AI-made reports.

Related papers

• Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data [62.63749675817477]
  AbdomenAtlas 2.0 is a dataset of 10,135 CT scans with a total of 15,130 tumor instances per-voxel manually annotated in six organs.<n>It achieves notable improvements over public datasets, with a +7% gain on DSC tests and +16% on out-of-distribution tests.
  arXiv  Detail & Related papers  (2025-10-16T16:08:09Z)
• Scaling Artificial Intelligence for Multi-Tumor Early Detection with More Reports, Fewer Masks [59.37427210144734]
  We introduce R-Super, which trains AI to segment tumors that match descriptions in medical reports.<n>When trained on 101,654 reports, AI models achieved performance comparable to those trained on 723 masks.<n>R-Super enabled segmentation of tumors in the spleen, gallbladder, prostate, bladder, uterus, and esophagus.
  arXiv  Detail & Related papers  (2025-10-16T15:35:44Z)
• ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reports [23.716614736159034]
  We introduce ReXGroundingCT, the first publicly available dataset linking free-text findings to pixel-level 3D segmentations in chest CT scans.<n>The dataset includes 3,142 non-contrast chest CT scans paired with standardized radiology reports from CT-RATE.<n>About 79% of findings are focal abnormalities and 21% are non-focal.
  arXiv  Detail & Related papers  (2025-07-29T17:27:15Z)
• Learning Segmentation from Radiology Reports [46.073392569125375]
  Tumor segmentation in CT scans is key for diagnosis, surgery, and prognosis.<n>Yet segmentation masks are scarce because their creation requires time and expertise.<n>We propose a report-supervision loss (R-Super) that converts radiology reports into voxel-wise supervision for tumor segmentation AI.
  arXiv  Detail & Related papers  (2025-07-08T01:37:34Z)
• PanTS: The Pancreatic Tumor Segmentation Dataset [49.32814895560867]
  PanTS is a large-scale, multi-institutional dataset curated to advance research in pancreatic CT analysis.<n>It contains 36,390 CT scans from 145 medical centers, with expert-validated, voxel-wise annotations of over 993,000 anatomical structures.
  arXiv  Detail & Related papers  (2025-07-02T02:10:46Z)
• FreeTumor: Large-Scale Generative Tumor Synthesis in Computed Tomography Images for Improving Tumor Recognition [11.984311048958318]
  FreeTumor is an innovative Generative AI (GAI) framework to enable large-scale tumor synthesis for mitigating data scarcity. We create the largest training dataset for tumor synthesis and recognition by curating 161,310 publicly available Computed Tomography (CT) volumes. To validate the fidelity of synthetic tumors, we engaged 13 board-certified radiologists in a Visual Turing Test to discern between synthetic and real tumors.
  arXiv  Detail & Related papers  (2025-02-23T07:00:09Z)
• Text-Driven Tumor Synthesis [28.654516965292444]
  Tumor synthesis can generate examples that AI often misses or over-detects.<n>Existing synthesis methods lack controllability over specific tumor characteristics.<n>We propose a new text-driven tumor synthesis approach, called TextoMorph.
  arXiv  Detail & Related papers  (2024-12-24T18:43:09Z)
• 3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models [51.855377054763345]
  This paper introduces 3D-CT-GPT, a Visual Question Answering (VQA)-based medical visual language model for generating radiology reports from 3D CT scans. Experiments on both public and private datasets demonstrate that 3D-CT-GPT significantly outperforms existing methods in terms of report accuracy and quality.
  arXiv  Detail & Related papers  (2024-09-28T12:31:07Z)
• Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation [42.06416052431378]
  2D radiology captioning is incompetent to reflect the real-world diagnostic challenge in the volumetric 3D anatomy. We collected an 18,885 text-scan pairs 3D-BrainCT dataset and applied clinical visual instruction tuning to train BrainGPT models to generate radiology-adherent 3D brain CT reports. Our work embodies a holistic framework that showcased the first-hand experience of curating a 3D brain CT dataset, fine-tuning anatomy-sensible language models, and proposing robust radiology evaluation metrics.
  arXiv  Detail & Related papers  (2024-07-02T12:58:35Z)
• RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis [56.57177181778517]
  RadGenome-Chest CT is a large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. We leverage the latest powerful universal segmentation and large language models to extend the original datasets.
  arXiv  Detail & Related papers  (2024-04-25T17:11:37Z)
• From Pixel to Cancer: Cellular Automata in Computed Tomography [12.524228287083888]
  Tumor synthesis seeks to create artificial tumors in medical images. This paper establishes a set of generic rules to simulate tumor development. We integrate the tumor state into the original computed tomography (CT) images to generate synthetic tumors across different organs.
  arXiv  Detail & Related papers  (2024-03-11T06:46:31Z)
• Towards Generalizable Tumor Synthesis [48.45704270448412]
  Tumor synthesis enables the creation of artificial tumors in medical images, facilitating the training of AI models for tumor detection and segmentation. This paper made a progressive stride toward generalizable tumor synthesis by leveraging a critical observation. We have ascertained that generative AI models, e.g., Diffusion Models, can create realistic tumors generalized to a range of organs even when trained on a limited number of tumor examples from only one organ.
  arXiv  Detail & Related papers  (2024-02-29T18:57:39Z)
• Glioblastoma Tumor Segmentation using an Ensemble of Vision Transformers [0.0]
  Glioblastoma is one of the most aggressive and deadliest types of brain cancer. Brain Radiology Aided by Intelligent Neural NETworks (BRAINNET) generates robust tumor segmentation maks.
  arXiv  Detail & Related papers  (2023-11-09T18:55:27Z)
• Complex Organ Mask Guided Radiology Report Generation [13.96983438709763]
  We propose the Complex Organ Mask Guided (termed as COMG) report generation model. We leverage prior knowledge of the disease corresponding to each organ in the fusion process to enhance the disease identification phase. Results on two public datasets show that COMG achieves a 11.4% and 9.7% improvement in terms of BLEU@4 scores over the SOTA model KiUT.
  arXiv  Detail & Related papers  (2023-11-04T05:34:24Z)
• Act Like a Radiologist: Radiology Report Generation across Anatomical Regions [50.13206214694885]
  X-RGen is a radiologist-minded report generation framework across six anatomical regions. In X-RGen, we seek to mimic the behaviour of human radiologists, breaking them down into four principal phases. We enhance the recognition capacity of the image encoder by analysing images and reports across various regions.
  arXiv  Detail & Related papers  (2023-05-26T07:12:35Z)
• CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans [45.83431075462771]
  Human readers or radiologists routinely perform full-body multi-organ multi-disease detection and diagnosis in clinical practice. Most medical AI systems are built to focus on single organs with a narrow list of a few diseases. CancerUniT is a query-based Mask Transformer model with the output of multi-tumor prediction.
  arXiv  Detail & Related papers  (2023-01-28T20:09:34Z)
• Medical Image Captioning via Generative Pretrained Transformers [57.308920993032274]
  We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO.
  arXiv  Detail & Related papers  (2022-09-28T10:27:10Z)
• EMT-NET: Efficient multitask network for computer-aided diagnosis of breast cancer [58.720142291102135]
  We propose an efficient and light-weighted learning architecture to classify and segment breast tumors simultaneously. We incorporate a segmentation task into a tumor classification network, which makes the backbone network learn representations focused on tumor regions. The accuracy, sensitivity, and specificity of tumor classification is 88.6%, 94.1%, and 85.3%, respectively.
  arXiv  Detail & Related papers  (2022-01-13T05:24:40Z)
• Ensemble CNN Networks for GBM Tumors Segmentation using Multi-parametric MRI [0.0]
  We propose a new aggregation of two deep learning frameworks namely, DeepSeg and nnU-Net for automatic glioblastoma recognition in pre-operative mpMRI. Our ensemble method obtains Dice similarity scores of 92.00, 87.33, and 84.10 and Hausdorff Distances of 3.81, 8.91, and 16.02 for the enhancing tumor, tumor core, and whole tumor regions.
  arXiv  Detail & Related papers  (2021-12-13T10:51:20Z)
• Segmentation for Classification of Screening Pancreatic Neuroendocrine Tumors [72.65802386845002]
  This work presents comprehensive results to detect in the early stage the pancreatic neuroendocrine tumors (PNETs) in abdominal CT scans. To the best of our knowledge, this task has not been studied before as a computational task. Our approach outperforms state-of-the-art segmentation networks and achieves a sensitivity of $89.47%$ at a specificity of $81.08%$.
  arXiv  Detail & Related papers  (2020-04-04T21:21:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.