Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis
- URL: http://arxiv.org/abs/2002.05000v1
- Date: Tue, 11 Feb 2020 08:26:42 GMT
- Title: Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis
- Authors: Tao Zhou, Huazhu Fu, Geng Chen, Jianbing Shen, and Ling Shao
- Abstract summary: We propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis.
In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality.
A multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality.
- Score: 143.55901940771568
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that
can provide images of different contrasts (i.e., modalities). Fusing this
multi-modal data has proven particularly effective for boosting model
performance in many tasks. However, due to poor data quality and frequent
patient dropout, collecting all modalities for every patient remains a
challenge. Medical image synthesis has been proposed as an effective solution
to this, where any missing modalities are synthesized from the existing ones.
In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for
multi-modal MR image synthesis, which learns a mapping from multi-modal source
images (i.e., existing modalities) to target images (i.e., missing modalities).
In our Hi-Net, a modality-specific network is utilized to learn representations
for each individual modality, and a fusion network is employed to learn the
common latent representation of multi-modal data. Then, a multi-modal synthesis
network is designed to densely combine the latent representation with
hierarchical features from each modality, acting as a generator to synthesize
the target images. Moreover, a layer-wise multi-modal fusion strategy is
presented to effectively exploit the correlations among multiple modalities, in
which a Mixed Fusion Block (MFB) is proposed to adaptively weight different
fusion strategies (i.e., element-wise summation, product, and maximization).
Extensive experiments demonstrate that the proposed model outperforms other
state-of-the-art medical image synthesis methods.
Related papers
- Disentangled Multimodal Brain MR Image Translation via Transformer-based
Modality Infuser [12.402947207350394]
We propose a transformer-based modality infuser designed to synthesize multimodal brain MR images.
In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features.
We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities.
arXiv Detail & Related papers (2024-02-01T06:34:35Z) - Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical
Representations [34.821129614819604]
We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities.
Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation.
Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for missing images.
arXiv Detail & Related papers (2023-09-15T20:21:03Z) - Unified Multi-Modal Image Synthesis for Missing Modality Imputation [23.681228202899984]
We propose a novel unified multi-modal image synthesis method for missing modality imputation.
The proposed method is effective in handling various synthesis tasks and shows superior performance compared to previous methods.
arXiv Detail & Related papers (2023-04-11T16:59:15Z) - CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI
Synthesis [11.803971719704721]
Most diffusion-based MRI synthesis models are using a single modality.
We propose the first diffusion-based multi-modality MRI synthesis model, namely Conditioned Latent Diffusion Model (CoLa-Diff)
Our experiments demonstrate that CoLa-Diff outperforms other state-of-the-art MRI synthesis methods.
arXiv Detail & Related papers (2023-03-24T15:46:10Z) - A Novel Unified Conditional Score-based Generative Framework for
Multi-modal Medical Image Completion [54.512440195060584]
We propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM)
UMM-CSGM employs a novel multi-in multi-out Conditional Score Network (mm-CSN) to learn a comprehensive set of cross-modal conditional distributions.
Experiments on BraTS19 dataset show that the UMM-CSGM can more reliably synthesize the heterogeneous enhancement and irregular area in tumor-induced lesions.
arXiv Detail & Related papers (2022-07-07T16:57:21Z) - A Learnable Variational Model for Joint Multimodal MRI Reconstruction
and Synthesis [4.056490719080639]
We propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI.
The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality.
arXiv Detail & Related papers (2022-04-08T01:35:19Z) - Multimodal Image Synthesis and Editing: The Generative AI Era [131.9569600472503]
multimodal image synthesis and editing has become a hot research topic in recent years.
We comprehensively contextualize the advance of the recent multimodal image synthesis and editing.
We describe benchmark datasets and evaluation metrics as well as corresponding experimental results.
arXiv Detail & Related papers (2021-12-27T10:00:16Z) - Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality.
In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network.
Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
arXiv Detail & Related papers (2021-10-15T13:16:59Z) - Multimodal Face Synthesis from Visual Attributes [85.87796260802223]
We propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images.
multimodal stretch-in modules are introduced in the discriminator which discriminates between real and fake images.
arXiv Detail & Related papers (2021-04-09T13:47:23Z) - Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement
and Gated Fusion [71.87627318863612]
We propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities.
Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code.
We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset.
arXiv Detail & Related papers (2020-02-22T14:32:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.