Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning

• URL: http://arxiv.org/abs/2503.01265v1
• Date: Mon, 03 Mar 2025 07:44:28 GMT
• Title: Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning
• Authors: Linhao Li, Changhui Su, Yu Guo, Huimao Zhang, Dong Liang, Kun Shang,
• Abstract summary: We propose a novel Transformer with Localization Prompts framework for synthesizing CE-MRI from non-contrast MR images.<n>Our architecture introduces three key innovations: a hierarchical backbone that uses efficient Transformer to process multi-scale features; a multi-stage fusion system that hierarchically integrate complementary information via spatial attention operations and cross-attention mechanisms, respectively.<n>The framework enables interactive clinical integration by allowing radiologists to input diagnostic prompts during inference, synergizing artificial intelligence with medical expertise.
• Score: 6.716077690014641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrast-enhanced magnetic resonance imaging (CE-MRI) is crucial for tumor detection and diagnosis, but the use of gadolinium-based contrast agents (GBCAs) in clinical settings raises safety concerns due to potential health risks. To circumvent these issues while preserving diagnostic accuracy, we propose a novel Transformer with Localization Prompts (TLP) framework for synthesizing CE-MRI from non-contrast MR images. Our architecture introduces three key innovations: a hierarchical backbone that uses efficient Transformer to process multi-scale features; a multi-stage fusion system consisting of Local and Global Fusion modules that hierarchically integrate complementary information via spatial attention operations and cross-attention mechanisms, respectively; and a Fuzzy Prompt Generation (FPG) module that enhances the TLP model's generalization by emulating radiologists' manual annotation through stochastic feature perturbation. The framework uniquely enables interactive clinical integration by allowing radiologists to input diagnostic prompts during inference, synergizing artificial intelligence with medical expertise. This research establishes a new paradigm for contrast-free MRI synthesis while addressing critical clinical needs for safer diagnostic procedures. Codes are available at https://github.com/ChanghuiSu/TLP.

Related papers

• CBM-RAG: Demonstrating Enhanced Interpretability in Radiology Report Generation with Multi-Agent RAG and Concept Bottleneck Models [1.7042756021131187]
  This paper presents an automated radiology report generation framework that combines Concept Bottleneck Models (CBMs) with a Multi-Agent Retrieval-Augmented Generation (RAG) system. CBMs map chest X-ray features to human-understandable clinical concepts, enabling transparent disease classification. RAG system integrates multi-agent collaboration and external knowledge to produce contextually rich, evidence-based reports.
  arXiv  Detail & Related papers  (2025-04-29T16:14:55Z)
• XLSTM-HVED: Cross-Modal Brain Tumor Segmentation and MRI Reconstruction Method Using Vision XLSTM and Heteromodal Variational Encoder-Decoder [9.141615533517719]
  We introduce the XLSTM-HVED model, which integrates a heteromodal encoder-decoder framework with the Vision XLSTM module to reconstruct missing MRI modalities.<n>Key innovation of our approach is the Self-Attention Variational (SAVE) module, which improves the integration of modal features.<n>Our experiments using the BraTS 2024 dataset demonstrate that our model significantly outperforms existing advanced methods in handling cases where modalities are missing.
  arXiv  Detail & Related papers  (2024-12-09T09:04:02Z)
• Clinical Evaluation of Medical Image Synthesis: A Case Study in Wireless Capsule Endoscopy [63.39037092484374]
  Synthetic Data Generation based on Artificial Intelligence (AI) can transform the way clinical medicine is delivered. This study focuses on the clinical evaluation of medical SDG, with a proof-of-concept investigation on diagnosing Inflammatory Bowel Disease (IBD) using Wireless Capsule Endoscopy (WCE) images. The results show that TIDE-II generates clinically plausible, very realistic WCE images, of improved quality compared to relevant state-of-the-art generative models.
  arXiv  Detail & Related papers  (2024-10-31T19:48:50Z)
• MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report [4.340464264725625]
  We introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs) and radiology/cardiology reports. We utilize LoRA-Peft to significantly reduce trainable parameters in the LLM and incorporate recent linear attention dropping strategy in the Vision Transformer(ViT) for smoother attention. To the best of our knowledge, we are the first to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report with this approach.
  arXiv  Detail & Related papers  (2024-10-21T17:42:41Z)
• Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation [51.28453192441364]
  Multimodal brain magnetic resonance (MR) imaging is indispensable in neuroscience and neurology. Current MR image synthesis approaches are typically trained on independent datasets for specific tasks. We present TUMSyn, a Text-guided Universal MR image Synthesis model, which can flexibly generate brain MR images.
  arXiv  Detail & Related papers  (2024-09-25T11:14:47Z)
• A Unified Framework for Synthesizing Multisequence Brain MRI via Hybrid Fusion [4.47838172826189]
  We propose a novel unified framework for synthesizing multisequence MR images, called Hybrid Fusion GAN (HF-GAN) We introduce a hybrid fusion encoder designed to ensure the disentangled extraction of complementary and modality-specific information. Common feature representations are transformed into a target latent space via the modality infuser to synthesize missing MR sequences.
  arXiv  Detail & Related papers  (2024-06-21T08:06:00Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• K-Space-Aware Cross-Modality Score for Synthesized Neuroimage Quality Assessment [71.27193056354741]
  The problem of how to assess cross-modality medical image synthesis has been largely unexplored. We propose a new metric K-CROSS to spur progress on this challenging problem. K-CROSS uses a pre-trained multi-modality segmentation network to predict the lesion location.
  arXiv  Detail & Related papers  (2023-07-10T01:26:48Z)
• Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution [55.52779466954026]
  Multi-contrast super-resolution (SR) reconstruction is promising to yield SR images with higher quality. Existing methods lack effective mechanisms to match and fuse these features for better reconstruction. We propose a novel network to address these problems by developing a set of innovative Transformer-empowered multi-scale contextual matching and aggregation techniques.
  arXiv  Detail & Related papers  (2022-03-26T01:42:59Z)
• ResViT: Residual vision transformers for multi-modal medical image synthesis [0.0]
  We propose a novel generative adversarial approach for medical image synthesis, ResViT, to combine local precision of convolution operators with contextual sensitivity of vision transformers. Our results indicate the superiority of ResViT against competing methods in terms of qualitative observations and quantitative metrics.
  arXiv  Detail & Related papers  (2021-06-30T12:57:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.