Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical
Representations
- URL: http://arxiv.org/abs/2309.08747v2
- Date: Tue, 19 Sep 2023 15:48:49 GMT
- Title: Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical
Representations
- Authors: Reuben Dorent, Nazim Haouchine, Fryderyk K\"ogl, Samuel Joutard,
Parikshit Juvekar, Erickson Torio, Alexandra Golby, Sebastien Ourselin, Sarah
Frisken, Tom Vercauteren, Tina Kapur, William M. Wells
- Abstract summary: We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that synthesizes missing images from various modalities.
Extending multi-modal VAEs with a hierarchical latent structure, we introduce a probabilistic formulation for fusing multi-modal images in a common latent representation.
Our model outperformed multi-modal VAEs, conditional GANs, and the current state-of-the-art unified method (ResViT) for missing images.
- Score: 34.821129614819604
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that
synthesizes missing images from various modalities. Extending multi-modal VAEs
with a hierarchical latent structure, we introduce a probabilistic formulation
for fusing multi-modal images in a common latent representation while having
the flexibility to handle incomplete image sets as input. Moreover, adversarial
learning is employed to generate sharper images. Extensive experiments are
performed on the challenging problem of joint intra-operative ultrasound (iUS)
and Magnetic Resonance (MR) synthesis. Our model outperformed multi-modal VAEs,
conditional GANs, and the current state-of-the-art unified method (ResViT) for
synthesizing missing images, demonstrating the advantage of using a
hierarchical latent representation and a principled probabilistic fusion
operation. Our code is publicly available
\url{https://github.com/ReubenDo/MHVAE}.
Related papers
- Disentangled Multimodal Brain MR Image Translation via Transformer-based
Modality Infuser [12.402947207350394]
We propose a transformer-based modality infuser designed to synthesize multimodal brain MR images.
In our method, we extract modality-agnostic features from the encoder and then transform them into modality-specific features.
We carried out experiments on the BraTS 2018 dataset, translating between four MR modalities.
arXiv Detail & Related papers (2024-02-01T06:34:35Z) - CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI
Synthesis [11.803971719704721]
Most diffusion-based MRI synthesis models are using a single modality.
We propose the first diffusion-based multi-modality MRI synthesis model, namely Conditioned Latent Diffusion Model (CoLa-Diff)
Our experiments demonstrate that CoLa-Diff outperforms other state-of-the-art MRI synthesis methods.
arXiv Detail & Related papers (2023-03-24T15:46:10Z) - Semantic Image Synthesis via Diffusion Models [159.4285444680301]
Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.
Recent work on semantic image synthesis mainly follows the emphde facto Generative Adversarial Nets (GANs)
arXiv Detail & Related papers (2022-06-30T18:31:51Z) - Image Generation with Multimodal Priors using Denoising Diffusion
Probabilistic Models [54.1843419649895]
A major challenge in using generative models to accomplish this task is the lack of paired data containing all modalities and corresponding outputs.
We propose a solution based on a denoising diffusion probabilistic synthesis models to generate images under multi-model priors.
arXiv Detail & Related papers (2022-06-10T12:23:05Z) - A Learnable Variational Model for Joint Multimodal MRI Reconstruction
and Synthesis [4.056490719080639]
We propose a novel deep-learning model for joint reconstruction and synthesis of multi-modal MRI.
The output of our model includes reconstructed images of the source modalities and high-quality image synthesized in the target modality.
arXiv Detail & Related papers (2022-04-08T01:35:19Z) - Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality.
In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network.
Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
arXiv Detail & Related papers (2021-10-15T13:16:59Z) - Multimodal Face Synthesis from Visual Attributes [85.87796260802223]
We propose a novel generative adversarial network that simultaneously synthesizes identity preserving multimodal face images.
multimodal stretch-in modules are introduced in the discriminator which discriminates between real and fake images.
arXiv Detail & Related papers (2021-04-09T13:47:23Z) - Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood
Estimation [54.17177006826262]
We develop a new generic conditional image synthesis method based on Implicit Maximum Likelihood Estimation (IMLE)
We demonstrate improved multimodal image synthesis performance on two tasks, single image super-resolution and image synthesis from scene layouts.
arXiv Detail & Related papers (2020-04-07T03:06:55Z) - Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis [143.55901940771568]
We propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis.
In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality.
A multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality.
arXiv Detail & Related papers (2020-02-11T08:26:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.