Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning

• URL: http://arxiv.org/abs/2503.01265v1
• Date: Mon, 03 Mar 2025 07:44:28 GMT
• Title: Interactive Gadolinium-Free MRI Synthesis: A Transformer with Localization Prompt Learning
• Abstract summary: We propose a novel Transformer with Localization Prompts framework for synthesizing CE-MRI from non-contrast MR images.<n>Our architecture introduces three key innovations: a hierarchical backbone that uses efficient Transformer to process multi-scale features; a multi-stage fusion system that hierarchically integrate complementary information via spatial attention operations and cross-attention mechanisms, respectively.<n>The framework enables interactive clinical integration by allowing radiologists to input diagnostic prompts during inference, synergizing artificial intelligence with medical expertise.
• Score: 6.716077690014641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrast-enhanced magnetic resonance imaging (CE-MRI) is crucial for tumor detection and diagnosis, but the use of gadolinium-based contrast agents (GBCAs) in clinical settings raises safety concerns due to potential health risks. To circumvent these issues while preserving diagnostic accuracy, we propose a novel Transformer with Localization Prompts (TLP) framework for synthesizing CE-MRI from non-contrast MR images. Our architecture introduces three key innovations: a hierarchical backbone that uses efficient Transformer to process multi-scale features; a multi-stage fusion system consisting of Local and Global Fusion modules that hierarchically integrate complementary information via spatial attention operations and cross-attention mechanisms, respectively; and a Fuzzy Prompt Generation (FPG) module that enhances the TLP model's generalization by emulating radiologists' manual annotation through stochastic feature perturbation. The framework uniquely enables interactive clinical integration by allowing radiologists to input diagnostic prompts during inference, synergizing artificial intelligence with medical expertise. This research establishes a new paradigm for contrast-free MRI synthesis while addressing critical clinical needs for safer diagnostic procedures. Codes are available at https://github.com/ChanghuiSu/TLP.

Related papers

• Concept-Grounded Reasoning with Prompt-Driven Localization for Interpretable Structured Report Generation [66.62662330558906]
  CORAL is a multimodal framework that integrates spatial grounding and concept-level supervision into a unified reasoning process.<n>CORAL employs a prompt-driven medical segmentation model to localize lesions and predicts multi-class clinical attributes through a Concept Bottleneck module.<n>Experiments on BUS-CoT and IU X-ray datasets demonstrate consistent improvements in diagnostic accuracy, concept consistency, and report quality over strong general-purpose and medical MLLMs.
  arXiv  Detail & Related papers  (2026-09-14T10:21:23Z)
• PhyDCM: A Reproducible Open-Source Framework for AI-Assisted Brain Tumor Classification from Multi-Sequence MRI [0.0]
  MRI-based medical imaging has become indispensable in modern clinical diagnosis.<n>Rapid growth in data volume poses challenges for conventional diagnostic approaches.<n>PhyDCM is introduced as an open-source software framework that integrates a hybrid classification architecture based on MedViT.<n>System is designed as a modular digital library that separates computational logic from the graphical interface.
  arXiv  Detail & Related papers  (2026-03-25T20:58:22Z)
• Any-to-All MRI Synthesis: A Unified Foundation Model for Nasopharyngeal Carcinoma and Its Downstream Applications [11.15455619035484]
  Traditional MRI synthesis methods are modality-specific, limited in anatomical adaptability, and lack clinical interpretability-failing to meet NPC's RT needs.<n>Here, we developed a unified foundation model integrating contrastive visual representation learning and vision-language alignment (VLA) to enable any-to-all MRI synthesis.<n>The model uses a contrastive encoder for modality-invariant representations and a CLIP-based text-informed decoder for semantically consistent synthesis, supporting any-to-all MRI synthesis via one unified foundation model.
  arXiv  Detail & Related papers  (2026-02-09T15:56:53Z)
• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• Transformers for Multimodal Brain State Decoding: Integrating Functional Magnetic Resonance Imaging Data and Medical Metadata [0.0]
  This paper presents a novel framework integrating transformer-based architectures with multimodal inputs, including fMRI data and DICOM metadata.<n>By employing attention mechanisms, the proposed method captures intricate spatial-temporal patterns and contextual relationships, enhancing model accuracy, interpretability, and robustness.<n>The potential robustness of this framework spans applications in clinical diagnostics, cognitive neuroscience, and personalized medicine.
  arXiv  Detail & Related papers  (2025-12-09T10:35:23Z)
• RTGMFF: Enhanced fMRI-based Brain Disorder Diagnosis via ROI-driven Text Generation and Multimodal Feature Fusion [13.589735978929085]
  We introduce RTGMFF, a framework that unifies automatic ROI-level text generation with multimodal feature fusion for brain-disorder diagnosis.<n>Tests show that RTGMFF surpasses current methods in diagnostic accuracy, achieving notable gains in sensitivity, specificity, and area under the ROC curve.
  arXiv  Detail & Related papers  (2025-09-03T11:05:57Z)
• NEARL-CLIP: Interacted Query Adaptation with Orthogonal Regularization for Medical Vision-Language Understanding [51.63264715941068]
  textbfNEARL-CLIP (iunderlineNteracted quunderlineEry underlineAdaptation with ounderlineRthogonaunderlineL Regularization) is a novel cross-modality interaction VLM-based framework.
  arXiv  Detail & Related papers  (2025-08-06T05:44:01Z)
• Graph-based Multi-Modal Interaction Lightweight Network for Brain Tumor Segmentation (GMLN-BTS) in Edge Iterative MRI Lesion Localization System (EdgeIMLocSys) [6.451534509235736]
  We propose the Edge Iterative MRI Lesion Localization System (EdgeIMLocSys), which integrates Continuous Learning from Human Feedback.<n>Central to this system is the Graph-based Multi-Modal Interaction Lightweight Network for Brain Tumor (GMLN-BTS)<n>Our proposed GMLN-BTS model achieves a Dice score of 85.1% on the BraTS 2017 dataset with only 4.58 million parameters, representing a 98% reduction compared to mainstream 3D Transformer models.
  arXiv  Detail & Related papers  (2025-07-14T07:29:49Z)
• RadFabric: Agentic AI System with Reasoning Capability for Radiology [61.25593938175618]
  RadFabric is a multi agent, multimodal reasoning framework that unifies visual and textual analysis for comprehensive CXR interpretation.<n>System employs specialized CXR agents for pathology detection, an Anatomical Interpretation Agent to map visual findings to precise anatomical structures, and a Reasoning Agent powered by large multimodal reasoning models to synthesize visual, anatomical, and clinical data into transparent and evidence based diagnoses.
  arXiv  Detail & Related papers  (2025-06-17T03:10:33Z)
• CBM-RAG: Demonstrating Enhanced Interpretability in Radiology Report Generation with Multi-Agent RAG and Concept Bottleneck Models [1.7042756021131187]
  This paper presents an automated radiology report generation framework that combines Concept Bottleneck Models (CBMs) with a Multi-Agent Retrieval-Augmented Generation (RAG) system. CBMs map chest X-ray features to human-understandable clinical concepts, enabling transparent disease classification. RAG system integrates multi-agent collaboration and external knowledge to produce contextually rich, evidence-based reports.
  arXiv  Detail & Related papers  (2025-04-29T16:14:55Z)
• XLSTM-HVED: Cross-Modal Brain Tumor Segmentation and MRI Reconstruction Method Using Vision XLSTM and Heteromodal Variational Encoder-Decoder [9.141615533517719]
  We introduce the XLSTM-HVED model, which integrates a heteromodal encoder-decoder framework with the Vision XLSTM module to reconstruct missing MRI modalities.<n>Key innovation of our approach is the Self-Attention Variational (SAVE) module, which improves the integration of modal features.<n>Our experiments using the BraTS 2024 dataset demonstrate that our model significantly outperforms existing advanced methods in handling cases where modalities are missing.
  arXiv  Detail & Related papers  (2024-12-09T09:04:02Z)
• Clinical Evaluation of Medical Image Synthesis: A Case Study in Wireless Capsule Endoscopy [63.39037092484374]
  Synthetic Data Generation based on Artificial Intelligence (AI) can transform the way clinical medicine is delivered. This study focuses on the clinical evaluation of medical SDG, with a proof-of-concept investigation on diagnosing Inflammatory Bowel Disease (IBD) using Wireless Capsule Endoscopy (WCE) images. The results show that TIDE-II generates clinically plausible, very realistic WCE images, of improved quality compared to relevant state-of-the-art generative models.
  arXiv  Detail & Related papers  (2024-10-31T19:48:50Z)
• MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report [4.340464264725625]
  We introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs) and radiology/cardiology reports. We utilize LoRA-Peft to significantly reduce trainable parameters in the LLM and incorporate recent linear attention dropping strategy in the Vision Transformer(ViT) for smoother attention. To the best of our knowledge, we are the first to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report with this approach.
  arXiv  Detail & Related papers  (2024-10-21T17:42:41Z)
• Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation [51.28453192441364]
  Multimodal brain magnetic resonance (MR) imaging is indispensable in neuroscience and neurology. Current MR image synthesis approaches are typically trained on independent datasets for specific tasks. We present TUMSyn, a Text-guided Universal MR image Synthesis model, which can flexibly generate brain MR images.
  arXiv  Detail & Related papers  (2024-09-25T11:14:47Z)
• A Unified Framework for Synthesizing Multisequence Brain MRI via Hybrid Fusion [4.47838172826189]
  We propose a novel unified framework for synthesizing multisequence MR images, called Hybrid Fusion GAN (HF-GAN) We introduce a hybrid fusion encoder designed to ensure the disentangled extraction of complementary and modality-specific information. Common feature representations are transformed into a target latent space via the modality infuser to synthesize missing MR sequences.
  arXiv  Detail & Related papers  (2024-06-21T08:06:00Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• An Interpretable Cross-Attentive Multi-modal MRI Fusion Framework for Schizophrenia Diagnosis [46.58592655409785]
  We propose a novel Cross-Attentive Multi-modal Fusion framework (CAMF) to capture both intra-modal and inter-modal relationships between fMRI and sMRI. Our approach significantly improves classification accuracy, as demonstrated by our evaluations on two extensive multi-modal brain imaging datasets. The gradient-guided Score-CAM is applied to interpret critical functional networks and brain regions involved in schizophrenia.
  arXiv  Detail & Related papers  (2024-03-29T20:32:30Z)
• Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading [47.50733518140625]
  Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. We propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading.
  arXiv  Detail & Related papers  (2024-01-17T07:54:49Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• K-Space-Aware Cross-Modality Score for Synthesized Neuroimage Quality Assessment [71.27193056354741]
  The problem of how to assess cross-modality medical image synthesis has been largely unexplored. We propose a new metric K-CROSS to spur progress on this challenging problem. K-CROSS uses a pre-trained multi-modality segmentation network to predict the lesion location.
  arXiv  Detail & Related papers  (2023-07-10T01:26:48Z)
• CKD-TransBTS: Clinical Knowledge-Driven Hybrid Transformer with Modality-Correlated Cross-Attention for Brain Tumor Segmentation [37.39921484146194]
  Brain tumor segmentation in magnetic resonance image (MRI) is crucial for brain tumor diagnosis, cancer management and research purposes. With the great success of the ten-year BraTS challenges, a lot of outstanding BTS models have been proposed to tackle the difficulties of BTS in different technical aspects. We propose a clinical knowledge-driven brain tumor segmentation model, called CKD-TransBTS.
  arXiv  Detail & Related papers  (2022-07-15T09:35:29Z)
• Transformer-empowered Multi-scale Contextual Matching and Aggregation for Multi-contrast MRI Super-resolution [55.52779466954026]
  Multi-contrast super-resolution (SR) reconstruction is promising to yield SR images with higher quality. Existing methods lack effective mechanisms to match and fuse these features for better reconstruction. We propose a novel network to address these problems by developing a set of innovative Transformer-empowered multi-scale contextual matching and aggregation techniques.
  arXiv  Detail & Related papers  (2022-03-26T01:42:59Z)
• ResViT: Residual vision transformers for multi-modal medical image synthesis [0.0]
  We propose a novel generative adversarial approach for medical image synthesis, ResViT, to combine local precision of convolution operators with contextual sensitivity of vision transformers. Our results indicate the superiority of ResViT against competing methods in terms of qualitative observations and quantitative metrics.
  arXiv  Detail & Related papers  (2021-06-30T12:57:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.