Explainable unsupervised multi-modal image registration using deep networks

• URL: http://arxiv.org/abs/2308.01994v1
• Date: Thu, 3 Aug 2023 19:13:48 GMT
• Title: Explainable unsupervised multi-modal image registration using deep networks
• Authors: Chengjia Wang, Giorgos Papanastasiou
• Abstract summary: MRI image registration aims to geometrically 'pair' diagnoses from different modalities, time points and slices. In this work, we show that our DL model becomes fully explainable, setting the framework to generalise our approach on further medical imaging data.
• Score: 2.197364252030876
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Clinical decision making from magnetic resonance imaging (MRI) combines complementary information from multiple MRI sequences (defined as 'modalities'). MRI image registration aims to geometrically 'pair' diagnoses from different modalities, time points and slices. Both intra- and inter-modality MRI registration are essential components in clinical MRI settings. Further, an MRI image processing pipeline that can address both afine and non-rigid registration is critical, as both types of deformations may be occuring in real MRI data scenarios. Unlike image classification, explainability is not commonly addressed in image registration deep learning (DL) methods, as it is challenging to interpet model-data behaviours against transformation fields. To properly address this, we incorporate Grad-CAM-based explainability frameworks in each major component of our unsupervised multi-modal and multi-organ image registration DL methodology. We previously demonstrated that we were able to reach superior performance (against the current standard Syn method). In this work, we show that our DL model becomes fully explainable, setting the framework to generalise our approach on further medical imaging data.

Related papers

• Towards General Text-guided Image Synthesis for Customized Multimodal Brain MRI Generation [51.28453192441364]
  Multimodal brain magnetic resonance (MR) imaging is indispensable in neuroscience and neurology. Current MR image synthesis approaches are typically trained on independent datasets for specific tasks. We present TUMSyn, a Text-guided Universal MR image Synthesis model, which can flexibly generate brain MR images.
  arXiv  Detail & Related papers  (2024-09-25T11:14:47Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• Dual Arbitrary Scale Super-Resolution for Multi-Contrast MRI [23.50915512118989]
  Multi-contrast Super-Resolution (SR) reconstruction is promising to yield SR images with higher quality. radiologists are accustomed to zooming the MR images at arbitrary scales rather than using a fixed scale. We propose an implicit neural representations based dual-arbitrary multi-contrast MRI super-resolution method, called Dual-ArbNet.
  arXiv  Detail & Related papers  (2023-07-05T14:43:26Z)
• On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis [58.634791552376235]
  Deep Learning (DL) models have achieved state-of-the-art performance in diagnosing multiple diseases using reconstructed images as input. DL models are sensitive to varying artifacts as it leads to changes in the input data distribution between the training and testing phases. We propose to use other normalization techniques, such as Group Normalization and Layer Normalization, to inject robustness into model performance against varying image artifacts.
  arXiv  Detail & Related papers  (2023-06-23T03:09:03Z)
• Model-Guided Multi-Contrast Deep Unfolding Network for MRI Super-resolution Reconstruction [68.80715727288514]
  We show how to unfold an iterative MGDUN algorithm into a novel model-guided deep unfolding network by taking the MRI observation matrix. In this paper, we propose a novel Model-Guided interpretable Deep Unfolding Network (MGDUN) for medical image SR reconstruction.
  arXiv  Detail & Related papers  (2022-09-15T03:58:30Z)
• Unsupervised Image Registration Towards Enhancing Performance and Explainability in Cardiac And Brain Image Analysis [3.5718941645696485]
  Inter- and intra-modality affine and non-rigid image registration is an essential medical image analysis process in clinical imaging. We present an un-supervised deep learning registration methodology which can accurately model affine and non-rigid trans-formations. Our methodology performs bi-directional cross-modality image synthesis to learn modality-invariant latent rep-resentations.
  arXiv  Detail & Related papers  (2022-03-07T12:54:33Z)
• Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation [75.58395328700821]
  We propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing.
  arXiv  Detail & Related papers  (2021-07-07T19:06:34Z)
• Deep Learning based Multi-modal Computing with Feature Disentanglement for MRI Image Synthesis [8.363448006582065]
  We propose a deep learning based multi-modal computing model for MRI synthesis with feature disentanglement strategy. The proposed approach decomposes each input modality into modality-invariant space with shared information and modality-specific space with specific information. To address the lack of specific information of the target modality in the test phase, a local adaptive fusion (LAF) module is adopted to generate a modality-like pseudo-target.
  arXiv  Detail & Related papers  (2021-05-06T17:22:22Z)
• Patch-based field-of-view matching in multi-modal images for electroporation-based ablations [0.6285581681015912]
  Multi-modal imaging sensors are currently involved at different steps of an interventional therapeutic work-flow. Merging this information relies on a correct spatial alignment of the observed anatomy between the acquired images. We show that a regional registration approach using voxel patches provides a good structural compromise between the voxel-wise and "global shifts" approaches.
  arXiv  Detail & Related papers  (2020-11-09T11:27:45Z)
• CoMIR: Contrastive Multimodal Image Representation for Registration [4.543268895439618]
  We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations) CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures.
  arXiv  Detail & Related papers  (2020-06-11T10:51:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.