MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion

• URL: http://arxiv.org/abs/2607.28565v1
• Date: Thu, 30 Jul 2026 17:30:21 GMT
• Title: MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion
• Abstract summary: Medical image fusion aims to integrate complementary information from diverse imaging modalities to support clinical diagnosis.<n>We propose MIND, a Multimodal Intent-Driven Network via Diffusion Transformers (DiTs) for medical image fusion.<n>We show that MIND delivers superior fusion quality, significantly improves downstream brain tumor segmentation accuracy, and enables flexible interactive fusion.
• Score: 9.316249658450124
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical image fusion aims to integrate complementary information from diverse imaging modalities to support clinical diagnosis. Existing methods typically apply uniform fusion rules globally, lacking a deep understanding of diagnostic intents and pathological structures. To address these limitations, we propose MIND, a Multimodal Intent-Driven Network via Diffusion Transformers (DiTs) for medical image fusion. Specifically, we utilize BioMedGPT to generate intent-driven fusion texts from source images, guiding the fusion process with pathology-aware diagnostic intents. To combat the loss of 2D spatial continuity caused by 1D sequence flattening in DiTs, we design a Multi-scale Latent Adapter. This module explicitly extracts source image features before serialization, injecting them into the network via strict dimensional alignment to effectively supplement image features. To resolve the semantic shift caused by decoupling image outputs from diagnostic intents, we design a medical semantic consistency loss. This loss ensures deep semantic locking between fused images and fusion texts while maintaining the stability of the underlying physical manifold reconstruction. Comprehensive experiments on the Harvard, BraTS, and GFP datasets reveal that MIND delivers superior fusion quality, significantly improves downstream brain tumor segmentation accuracy, and enables flexible interactive fusion, holding significant promise for intent-driven intelligent clinical decision support systems.

Related papers

• AMIF: Authorizable Medical Image Fusion Model with Built-in Authentication [61.10547481840624]
  A powerful multimodal image fusion model relies on high-quality, clinically representative multimodal training data.<n>Current image fusion models generate fused outputs without built-in mechanisms to safeguard intellectual property rights.<n>We propose AMIF, the first Authorizable Medical Image Fusion model with built-in authentication.
  arXiv  Detail & Related papers  (2026-03-25T13:34:22Z)
• Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers [57.54843029965778]
  Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail.<n>We propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer.<n>IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods.
  arXiv  Detail & Related papers  (2026-03-11T06:14:11Z)
• Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach [99.80480649258557]
  DiTFuse is an instruction-driven framework that performs semantics-aware fusion within a single model.<n>Experiments on public IVIF, MFF, and MEF benchmarks confirm superior quantitative and qualitative performance, sharper textures, and better semantic retention.
  arXiv  Detail & Related papers  (2025-12-08T05:04:54Z)
• DM-FNet: Unified multimodal medical image fusion via diffusion process-trained encoder-decoder [13.87371547830489]
  Multimodal medical image fusion (MMIF) extracts the most meaningful information from multiple source images.<n>Existing MMIF methods have limited capacity to capture detailed features during conventional training.<n>This study proposes a two-stage diffusion model-based fusion network (DM-FNet) to achieve unified MMIF.
  arXiv  Detail & Related papers  (2025-06-18T07:55:06Z)
• SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis [11.356721356096564]
  We propose a novel semantic-guided medical image fusion approach that incorporates medical prior knowledge into the fusion process.<n>We generate diagnostic reports from the fused images to assess the preservation of medical information.<n> Experimental results on test datasets demonstrate that the proposed method achieves superior performance in both qualitative and quantitative evaluations.
  arXiv  Detail & Related papers  (2025-05-18T06:15:00Z)
• FusionINN: Decomposable Image Fusion for Brain Tumor Monitoring [6.45135260209391]
  We introduce FusionINN, a novel decomposable image fusion framework. We are the first to investigate the decomposability of fused images. Our approach offers faster and qualitatively better fusion results.
  arXiv  Detail & Related papers  (2024-03-23T08:54:03Z)
• A New Multimodal Medical Image Fusion based on Laplacian Autoencoder with Channel Attention [3.1531360678320897]
  Deep learning models have achieved end-to-end image fusion with highly robust and accurate performance. Most DL-based fusion models perform down-sampling on the input images to minimize the number of learnable parameters and computations. We propose a new multimodal medical image fusion model is proposed that is based on integrated Laplacian-Gaussian concatenation with attention pooling.
  arXiv  Detail & Related papers  (2023-10-18T11:29:53Z)
• Three-Dimensional Medical Image Fusion with Deformable Cross-Attention [10.26573411162757]
  Multimodal medical image fusion plays an instrumental role in several areas of medical image processing. Traditional fusion methods tend to process each modality independently before combining the features and reconstructing the fusion image. In this study, we introduce an innovative unsupervised feature mutual learning fusion network designed to rectify these limitations.
  arXiv  Detail & Related papers  (2023-10-10T04:10:56Z)
• Equivariant Multi-Modality Image Fusion [124.11300001864579]
  We propose the Equivariant Multi-Modality imAge fusion paradigm for end-to-end self-supervised learning. Our approach is rooted in the prior knowledge that natural imaging responses are equivariant to certain transformations. Experiments confirm that EMMA yields high-quality fusion results for infrared-visible and medical images.
  arXiv  Detail & Related papers  (2023-05-19T05:50:24Z)
• MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer [53.575573940055335]
  We propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities.
  arXiv  Detail & Related papers  (2023-01-19T03:42:36Z)
• An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion [24.415389503712596]
  Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. We introduce a novel Dilated Residual Attention Network for the medical image fusion task.
  arXiv  Detail & Related papers  (2022-12-09T04:19:43Z)
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion [68.78897015832113]
  We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion.<n>Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
  arXiv  Detail & Related papers  (2022-11-20T12:02:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.