Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion
- URL: http://arxiv.org/abs/2403.03217v1
- Date: Tue, 5 Mar 2024 18:58:55 GMT
- Title: Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion
- Authors: Meng Zheng, Benjamin Planche, Xuan Gong, Fan Yang, Terrence Chen,
Ziyan Wu
- Abstract summary: 3D patient body modeling is critical to the success of automated patient positioning for smart medical scanning and operating rooms.
Existing CNN-based end-to-end patient modeling solutions typically require customized network designs demanding large amount of relevant training data.
We propose a generic modularized 3D patient modeling method consists of (a) a multi-modal keypoint detection module with attentive fusion for 2D patient joint localization.
We demonstrate the efficacy of the proposed method by extensive patient positioning experiments on both public and clinical data.
- Score: 32.71972792352939
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: 3D patient body modeling is critical to the success of automated patient
positioning for smart medical scanning and operating rooms. Existing CNN-based
end-to-end patient modeling solutions typically require a) customized network
designs demanding large amount of relevant training data, covering extensive
realistic clinical scenarios (e.g., patient covered by sheets), which leads to
suboptimal generalizability in practical deployment, b) expensive 3D human
model annotations, i.e., requiring huge amount of manual effort, resulting in
systems that scale poorly. To address these issues, we propose a generic
modularized 3D patient modeling method consists of (a) a multi-modal keypoint
detection module with attentive fusion for 2D patient joint localization, to
learn complementary cross-modality patient body information, leading to
improved keypoint localization robustness and generalizability in a wide
variety of imaging (e.g., CT, MRI etc.) and clinical scenarios (e.g., heavy
occlusions); and (b) a self-supervised 3D mesh regression module which does not
require expensive 3D mesh parameter annotations to train, bringing immediate
cost benefits for clinical deployment. We demonstrate the efficacy of the
proposed method by extensive patient positioning experiments on both public and
clinical data. Our evaluation results achieve superior patient positioning
performance across various imaging modalities in real clinical scenarios.
Related papers
- Inter-slice Super-resolution of Magnetic Resonance Images by Pre-training and Self-supervised Fine-tuning [49.197385954021456]
In clinical practice, 2D magnetic resonance (MR) sequences are widely adopted. While individual 2D slices can be stacked to form a 3D volume, the relatively large slice spacing can pose challenges for visualization and subsequent analysis tasks.
To reduce slice spacing, deep-learning-based super-resolution techniques are widely investigated.
Most current solutions require a substantial number of paired high-resolution and low-resolution images for supervised training, which are typically unavailable in real-world scenarios.
arXiv Detail & Related papers (2024-06-10T02:20:26Z) - Deep Learning-Based Brain Image Segmentation for Automated Tumour Detection [0.0]
The objective is to leverage state-of-the-art convolutional neural networks (CNNs) on a large dataset of brain MRI scans for segmentation.
The proposed methodology applies pre-processing techniques for enhanced performance and generalizability.
arXiv Detail & Related papers (2024-04-06T15:09:49Z) - Three-dimensional Bone Image Synthesis with Generative Adversarial
Networks [2.499907423888049]
This work demonstrates that three-dimensional generative adversarial networks (GANs) can be efficiently trained to generate high-resolution medical volumes.
GAN inversion is successfully implemented for the three-dimensional setting and used for extensive research on model interpretability.
arXiv Detail & Related papers (2023-10-26T08:08:17Z) - On the Localization of Ultrasound Image Slices within Point Distribution
Models [84.27083443424408]
Thyroid disorders are most commonly diagnosed using high-resolution Ultrasound (US)
Longitudinal tracking is a pivotal diagnostic protocol for monitoring changes in pathological thyroid morphology.
We present a framework for automated US image slice localization within a 3D shape representation.
arXiv Detail & Related papers (2023-09-01T10:10:46Z) - BOSS: Bones, Organs and Skin Shape Model [10.50175010474078]
We propose a deformable human shape and pose model that combines skin, internal organs, and bones, learned from CT images.
By modeling the statistical variations in a pose-normalized space using probabilistic PCA, our approach offers a holistic representation of the body.
arXiv Detail & Related papers (2023-03-08T22:31:24Z) - DeepAD: A Robust Deep Learning Model of Alzheimer's Disease Progression
for Real-World Clinical Applications [0.9999629695552196]
We propose a novel multi-task deep learning model to predict Alzheimer's disease progression.
Our model integrates high dimensional MRI features from a 3D convolutional neural network with other data modalities.
arXiv Detail & Related papers (2022-03-17T05:42:00Z) - Automated Model Design and Benchmarking of 3D Deep Learning Models for
COVID-19 Detection with Chest CT Scans [72.04652116817238]
We propose a differentiable neural architecture search (DNAS) framework to automatically search for the 3D DL models for 3D chest CT scans classification.
We also exploit the Class Activation Mapping (CAM) technique on our models to provide the interpretability of the results.
arXiv Detail & Related papers (2021-01-14T03:45:01Z) - Adversarial Sample Enhanced Domain Adaptation: A Case Study on
Predictive Modeling with Electronic Health Records [57.75125067744978]
We propose a data augmentation method to facilitate domain adaptation.
adversarially generated samples are used during domain adaptation.
Results confirm the effectiveness of our method and the generality on different tasks.
arXiv Detail & Related papers (2021-01-13T03:20:20Z) - Hierarchical Amortized Training for Memory-efficient High Resolution 3D
GAN [52.851990439671475]
We propose a novel end-to-end GAN architecture that can generate high-resolution 3D images.
We achieve this goal by using different configurations between training and inference.
Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation.
arXiv Detail & Related papers (2020-08-05T02:33:04Z) - Modelling the Distribution of 3D Brain MRI using a 2D Slice VAE [66.63629641650572]
We propose a method to model 3D MR brain volumes distribution by combining a 2D slice VAE with a Gaussian model that captures the relationships between slices.
We also introduce a novel evaluation method for generated volumes that quantifies how well their segmentations match those of true brain anatomy.
arXiv Detail & Related papers (2020-07-09T13:23:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.