Cross Modality 3D Navigation Using Reinforcement Learning and Neural
Style Transfer
- URL: http://arxiv.org/abs/2111.03485v1
- Date: Fri, 5 Nov 2021 13:11:45 GMT
- Title: Cross Modality 3D Navigation Using Reinforcement Learning and Neural
Style Transfer
- Authors: Cesare Magnetti, Hadrien Reynaud, Bernhard Kainz
- Abstract summary: This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging.
We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments.
Our framework does not require any labelled clinical data and integrates easily with several image translation techniques.
- Score: 3.0152753984876854
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to
perform navigation in 3D anatomical volumes from medical imaging. We utilize
Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym
environments and assess the generalization capabilities of our agents to
clinical CT volumes. Our framework does not require any labelled clinical data
and integrates easily with several image translation techniques, enabling cross
modality applications. Further, we solely condition our agents on 2D slices,
breaking grounds for 3D guidance in much more difficult imaging modalities,
such as ultrasound imaging. This is an important step towards user guidance
during the acquisition of standardised diagnostic view planes, improving
diagnostic consistency and facilitating better case comparison.
Related papers
- X-ray2CTPA: Generating 3D CTPA scans from 2D X-ray conditioning [24.233484690096898]
Chest X-rays or chest radiography (CXR) enables limited imaging compared to computed tomography (CT) scans.
CT scans entail higher costs, greater radiation exposure, and are less accessible than CXRs.
In this work we explore cross-modal translation from a 2D low contrast-resolution X-ray input to a 3D high contrast and spatial-resolutionA scan.
arXiv Detail & Related papers (2024-06-23T13:53:35Z) - CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios [53.94122089629544]
We introduce CT-GLIP (Grounded Language-Image Pretraining with CT scans), a novel method that constructs organ-level image-text pairs to enhance multimodal contrastive learning.
Our method, trained on a multimodal CT dataset comprising 44,011 organ-level vision-text pairs from 17,702 patients across 104 organs, demonstrates it can identify organs and abnormalities in a zero-shot manner using natural languages.
arXiv Detail & Related papers (2024-04-23T17:59:01Z) - Generative Enhancement for 3D Medical Images [74.17066529847546]
We propose GEM-3D, a novel generative approach to the synthesis of 3D medical images.
Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask.
By decomposing the 3D medical images into masks and patient prior information, GEM-3D offers a flexible yet effective solution for generating versatile 3D images.
arXiv Detail & Related papers (2024-03-19T15:57:04Z) - SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion
Classification Using 3D Multi-Phase Imaging [59.78761085714715]
This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification.
The proposed framework has been validated through comprehensive experiments on two clinical datasets.
To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
arXiv Detail & Related papers (2024-02-27T06:32:56Z) - On the Localization of Ultrasound Image Slices within Point Distribution
Models [84.27083443424408]
Thyroid disorders are most commonly diagnosed using high-resolution Ultrasound (US)
Longitudinal tracking is a pivotal diagnostic protocol for monitoring changes in pathological thyroid morphology.
We present a framework for automated US image slice localization within a 3D shape representation.
arXiv Detail & Related papers (2023-09-01T10:10:46Z) - Multi-View Vertebra Localization and Identification from CT Images [57.56509107412658]
We propose a multi-view vertebra localization and identification from CT images.
We convert the 3D problem into a 2D localization and identification task on different views.
Our method can learn the multi-view global information naturally.
arXiv Detail & Related papers (2023-07-24T14:43:07Z) - Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D
Brain MRI Synthesis [35.45013834475523]
Cross-modality medical image synthesis is a critical topic and has the potential to facilitate numerous applications in the medical imaging field.
Most current medical image synthesis methods rely on generative adversarial networks and suffer from notorious mode collapse and unstable training.
We introduce a new paradigm for volumetric medical data synthesis by leveraging 2D backbones and present a diffusion-based framework, Make-A-Volume.
arXiv Detail & Related papers (2023-07-19T16:01:09Z) - Multimodal Information Fusion for Glaucoma and DR Classification [1.5616442980374279]
Multimodal information is frequently available in medical tasks. By combining information from multiple sources, clinicians are able to make more accurate judgments.
Our paper investigates three multimodal information fusion strategies based on deep learning to solve retinal analysis tasks.
arXiv Detail & Related papers (2022-09-02T12:19:03Z) - Slice-level Detection of Intracranial Hemorrhage on CT Using Deep
Descriptors of Adjacent Slices [0.31317409221921133]
We propose a new strategy to train emphslice-level classifiers on CT scans based on the descriptors of the adjacent slices along the axis.
We obtain a single model in the top 4% best-performing solutions of the RSNA Intracranial Hemorrhage dataset challenge.
The proposed method is general and can be applied to other 3D medical diagnosis tasks such as MRI imaging.
arXiv Detail & Related papers (2022-08-05T23:20:37Z) - The entire network structure of Crossmodal Transformer [4.605531191013731]
The proposed approach first deep learns skeletal features from 2D X-ray and 3D CT images.
As a result, the well-trained network can directly predict the spatial correspondence between arbitrary 2D X-ray and 3D CT.
arXiv Detail & Related papers (2021-04-29T11:47:31Z) - Hierarchical Amortized Training for Memory-efficient High Resolution 3D
GAN [52.851990439671475]
We propose a novel end-to-end GAN architecture that can generate high-resolution 3D images.
We achieve this goal by using different configurations between training and inference.
Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation.
arXiv Detail & Related papers (2020-08-05T02:33:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.