Cross Modality 3D Navigation Using Reinforcement Learning and Neural Style Transfer

• URL: http://arxiv.org/abs/2111.03485v1
• Date: Fri, 5 Nov 2021 13:11:45 GMT
• Title: Cross Modality 3D Navigation Using Reinforcement Learning and Neural Style Transfer
• Authors: Cesare Magnetti, Hadrien Reynaud, Bernhard Kainz
• Abstract summary: This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques.
• Score: 3.0152753984876854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents the use of Multi-Agent Reinforcement Learning (MARL) to perform navigation in 3D anatomical volumes from medical imaging. We utilize Neural Style Transfer to create synthetic Computed Tomography (CT) agent gym environments and assess the generalization capabilities of our agents to clinical CT volumes. Our framework does not require any labelled clinical data and integrates easily with several image translation techniques, enabling cross modality applications. Further, we solely condition our agents on 2D slices, breaking grounds for 3D guidance in much more difficult imaging modalities, such as ultrasound imaging. This is an important step towards user guidance during the acquisition of standardised diagnostic view planes, improving diagnostic consistency and facilitating better case comparison.

Related papers

• Cross-D Conv: Cross-Dimensional Transferable Knowledge Base via Fourier Shifting Operation [3.69758875412828]
  Cross-D Conv operation bridges the dimensional gap by learning the phase shifting in the Fourier domain. Our method enables seamless weight transfer between 2D and 3D convolution operations, effectively facilitating cross-dimensional learning.
  arXiv  Detail & Related papers  (2024-11-02T13:03:44Z)
• 3D-CT-GPT: Generating 3D Radiology Reports through Integration of Large Vision-Language Models [51.855377054763345]
  This paper introduces 3D-CT-GPT, a Visual Question Answering (VQA)-based medical visual language model for generating radiology reports from 3D CT scans. Experiments on both public and private datasets demonstrate that 3D-CT-GPT significantly outperforms existing methods in terms of report accuracy and quality.
  arXiv  Detail & Related papers  (2024-09-28T12:31:07Z)
• Autoregressive Sequence Modeling for 3D Medical Image Representation [48.706230961589924]
  We introduce a pioneering method for learning 3D medical image representations through an autoregressive sequence pre-training framework. Our approach various 3D medical images based on spatial, contrast, and semantic correlations, treating them as interconnected visual tokens within a token sequence.
  arXiv  Detail & Related papers  (2024-09-13T10:19:10Z)
• X-ray2CTPA: Generating 3D CTPA scans from 2D X-ray conditioning [24.233484690096898]
  Chest X-rays or chest radiography (CXR) enables limited imaging compared to computed tomography (CT) scans. CT scans entail higher costs, greater radiation exposure, and are less accessible than CXRs. In this work we explore cross-modal translation from a 2D low contrast-resolution X-ray input to a 3D high contrast and spatial-resolutionA scan.
  arXiv  Detail & Related papers  (2024-06-23T13:53:35Z)
• Generative Enhancement for 3D Medical Images [74.17066529847546]
  We propose GEM-3D, a novel generative approach to the synthesis of 3D medical images. Our method begins with a 2D slice, noted as the informed slice to serve the patient prior, and propagates the generation process using a 3D segmentation mask. By decomposing the 3D medical images into masks and patient prior information, GEM-3D offers a flexible yet effective solution for generating versatile 3D images.
  arXiv  Detail & Related papers  (2024-03-19T15:57:04Z)
• On the Localization of Ultrasound Image Slices within Point Distribution Models [84.27083443424408]
  Thyroid disorders are most commonly diagnosed using high-resolution Ultrasound (US) Longitudinal tracking is a pivotal diagnostic protocol for monitoring changes in pathological thyroid morphology. We present a framework for automated US image slice localization within a 3D shape representation.
  arXiv  Detail & Related papers  (2023-09-01T10:10:46Z)
• Multi-View Vertebra Localization and Identification from CT Images [57.56509107412658]
  We propose a multi-view vertebra localization and identification from CT images. We convert the 3D problem into a 2D localization and identification task on different views. Our method can learn the multi-view global information naturally.
  arXiv  Detail & Related papers  (2023-07-24T14:43:07Z)
• Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis [35.45013834475523]
  Cross-modality medical image synthesis is a critical topic and has the potential to facilitate numerous applications in the medical imaging field. Most current medical image synthesis methods rely on generative adversarial networks and suffer from notorious mode collapse and unstable training. We introduce a new paradigm for volumetric medical data synthesis by leveraging 2D backbones and present a diffusion-based framework, Make-A-Volume.
  arXiv  Detail & Related papers  (2023-07-19T16:01:09Z)
• Slice-level Detection of Intracranial Hemorrhage on CT Using Deep Descriptors of Adjacent Slices [0.31317409221921133]
  We propose a new strategy to train emphslice-level classifiers on CT scans based on the descriptors of the adjacent slices along the axis. We obtain a single model in the top 4% best-performing solutions of the RSNA Intracranial Hemorrhage dataset challenge. The proposed method is general and can be applied to other 3D medical diagnosis tasks such as MRI imaging.
  arXiv  Detail & Related papers  (2022-08-05T23:20:37Z)
• The entire network structure of Crossmodal Transformer [4.605531191013731]
  The proposed approach first deep learns skeletal features from 2D X-ray and 3D CT images. As a result, the well-trained network can directly predict the spatial correspondence between arbitrary 2D X-ray and 3D CT.
  arXiv  Detail & Related papers  (2021-04-29T11:47:31Z)
• Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN [52.851990439671475]
  We propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation.
  arXiv  Detail & Related papers  (2020-08-05T02:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.