MedRegion-CT: Region-Focused Multimodal LLM for Comprehensive 3D CT Report Generation

• URL: http://arxiv.org/abs/2506.23102v1
• Date: Sun, 29 Jun 2025 06:08:55 GMT
• Title: MedRegion-CT: Region-Focused Multimodal LLM for Comprehensive 3D CT Report Generation
• Authors: Sunggu Kyung, Jinyoung Seo, Hyunseok Lim, Dongyeong Kim, Hyungbin Park, Jimin Sung, Jihyun Kim, Wooyoung Jo, Yoojin Nam, Namkug Kim,
• Abstract summary: We propose MedRegion-CT, a region-focused Multi-Modal Large Language Model (MLLM) framework, featuring three key innovations.<n>First, we introduce Region Representative ($R2$) Token Pooling, which utilizes a 2D-wise pretrained vision model to efficiently extract 3D CT features.<n>Second, a universal segmentation model generates pseudo-masks, which are then processed by a mask encoder to extract region-centric features.<n>Third, we leverage segmentation results to extract patient-specific attributions, including organ size, diameter, and locations.
• Score: 1.6515663221123749
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The recent release of RadGenome-Chest CT has significantly advanced CT-based report generation. However, existing methods primarily focus on global features, making it challenging to capture region-specific details, which may cause certain abnormalities to go unnoticed. To address this, we propose MedRegion-CT, a region-focused Multi-Modal Large Language Model (MLLM) framework, featuring three key innovations. First, we introduce Region Representative ($R^2$) Token Pooling, which utilizes a 2D-wise pretrained vision model to efficiently extract 3D CT features. This approach generates global tokens representing overall slice features and region tokens highlighting target areas, enabling the MLLM to process comprehensive information effectively. Second, a universal segmentation model generates pseudo-masks, which are then processed by a mask encoder to extract region-centric features. This allows the MLLM to focus on clinically relevant regions, using six predefined region masks. Third, we leverage segmentation results to extract patient-specific attributions, including organ size, diameter, and locations. These are converted into text prompts, enriching the MLLM's understanding of patient-specific contexts. To ensure rigorous evaluation, we conducted benchmark experiments on report generation using the RadGenome-Chest CT. MedRegion-CT achieved state-of-the-art performance, outperforming existing methods in natural language generation quality and clinical relevance while maintaining interpretability. The code for our framework is publicly available.

Related papers

• Large Language Model with Region-guided Referring and Grounding for CT Report Generation [4.804660464589285]
  Existing methods primarily only consider the global features of the entire volume.<n>We propose Reg2RG, the first region-guided referring and grounding framework for CT report generation.
  arXiv  Detail & Related papers  (2024-11-23T12:25:06Z)
• Improving 3D Medical Image Segmentation at Boundary Regions using Local Self-attention and Global Volume Mixing [14.0825980706386]
  Volumetric medical image segmentation is a fundamental problem in medical image analysis where the objective is to accurately classify a given 3D volumetric medical image with voxel-level precision. In this work, we propose a novel hierarchical encoder-decoder-based framework that strives to explicitly capture the local and global dependencies for 3D medical image segmentation. The proposed framework exploits local volume-based self-attention to encode the local dependencies at high resolution and introduces a novel volumetric-mixer to capture the global dependencies at low-resolution feature representations.
  arXiv  Detail & Related papers  (2024-10-20T11:08:38Z)
• Perspective+ Unet: Enhancing Segmentation with Bi-Path Fusion and Efficient Non-Local Attention for Superior Receptive Fields [19.71033340093199]
  We propose a novel architecture, Perspective+ Unet, to overcome limitations in medical image segmentation. The framework incorporates an efficient non-local transformer block, named ENLTB, which utilizes kernel function approximation for effective long-range dependency capture. Experimental results on the ACDC and datasets demonstrate the effectiveness of our proposed Perspective+ Unet.
  arXiv  Detail & Related papers  (2024-06-20T07:17:39Z)
• RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysis [56.57177181778517]
  RadGenome-Chest CT is a large-scale, region-guided 3D chest CT interpretation dataset based on CT-RATE. We leverage the latest powerful universal segmentation and large language models to extend the original datasets.
  arXiv  Detail & Related papers  (2024-04-25T17:11:37Z)
• A Mutual Inclusion Mechanism for Precise Boundary Segmentation in Medical Images [2.9137615132901704]
  We present a novel deep learning-based approach, MIPC-Net, for precise boundary segmentation in medical images. We introduce the MIPC module, which enhances the focus on channel information when extracting position features. We also propose the GL-MIPC-Residue, a global residual connection that enhances the integration of the encoder and decoder.
  arXiv  Detail & Related papers  (2024-04-12T02:14:35Z)
• Mask-Enhanced Segment Anything Model for Tumor Lesion Semantic Segmentation [48.107348956719775]
  We introduce Mask-Enhanced SAM (M-SAM), an innovative architecture tailored for 3D tumor lesion segmentation. We propose a novel Mask-Enhanced Adapter (MEA) within M-SAM that enriches the semantic information of medical images with positional data from coarse segmentation masks. Our M-SAM achieves high segmentation accuracy and also exhibits robust generalization.
  arXiv  Detail & Related papers  (2024-03-09T13:37:02Z)
• Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoders [52.66195794216989]
  We propose Point Feature Enhancement Masked Autoencoders (Point-FEMAE) to learn compact 3D representations. Point-FEMAE consists of a global branch and a local branch to capture latent semantic features. Our method significantly improves the pre-training efficiency compared to cross-modal alternatives.
  arXiv  Detail & Related papers  (2023-12-17T14:17:05Z)
• Multi-View Vertebra Localization and Identification from CT Images [57.56509107412658]
  We propose a multi-view vertebra localization and identification from CT images. We convert the 3D problem into a 2D localization and identification task on different views. Our method can learn the multi-view global information naturally.
  arXiv  Detail & Related papers  (2023-07-24T14:43:07Z)
• R-MAE: Regions Meet Masked Autoencoders [113.73147144125385]
  We explore regions as a potential visual analogue of words for self-supervised image representation learning. Inspired by Masked Autoencoding (MAE), a generative pre-training baseline, we propose masked region autoencoding to learn from groups of pixels or regions.
  arXiv  Detail & Related papers  (2023-06-08T17:56:46Z)
• A unified 3D framework for Organs at Risk Localization and Segmentation for Radiation Therapy Planning [56.52933974838905]
  Current medical workflow requires manual delineation of organs-at-risk (OAR) In this work, we aim to introduce a unified 3D pipeline for OAR localization-segmentation. Our proposed framework fully enables the exploitation of 3D context information inherent in medical imaging.
  arXiv  Detail & Related papers  (2022-03-01T17:08:41Z)
• PGL: Prior-Guided Local Self-supervised Learning for 3D Medical Image Segmentation [87.50205728818601]
  We propose a PriorGuided Local (PGL) self-supervised model that learns the region-wise local consistency in the latent feature space. Our PGL model learns the distinctive representations of local regions, and hence is able to retain structural information.
  arXiv  Detail & Related papers  (2020-11-25T11:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.