Modality-Projection Universal Model for Comprehensive Full-Body Medical Imaging Segmentation

• URL: http://arxiv.org/abs/2412.19026v1
• Date: Thu, 26 Dec 2024 02:23:27 GMT
• Title: Modality-Projection Universal Model for Comprehensive Full-Body Medical Imaging Segmentation
• Abstract summary: This study introduces and evaluate a Modality Projection Universal Model (MPUM)<n>MPUM employs a novel modality-projection strategy, which allows the model to dynamically adjust its parameters to optimize performance across different imaging modalities.<n>MPUM's unique controller-based convolution layer enables visualization of saliency maps across all network layers, significantly enhancing the model's interpretability.
• Score: 11.395596488197098
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The integration of deep learning in medical imaging has shown great promise for enhancing diagnostic, therapeutic, and research outcomes. However, applying universal models across multiple modalities remains challenging due to the inherent variability in data characteristics. This study aims to introduce and evaluate a Modality Projection Universal Model (MPUM). MPUM employs a novel modality-projection strategy, which allows the model to dynamically adjust its parameters to optimize performance across different imaging modalities. The MPUM demonstrated superior accuracy in identifying anatomical structures, enabling precise quantification for improved clinical decision-making. It also identifies metabolic associations within the brain-body axis, advancing research on brain-body physiological correlations. Furthermore, MPUM's unique controller-based convolution layer enables visualization of saliency maps across all network layers, significantly enhancing the model's interpretability.

Related papers

• Data-Efficient Meningioma Segmentation via Implicit Spatiotemporal Mixing and Sim2Real Semantic Injection [6.992254817538211]
  We propose a novel dual-augmentation framework that integrates spatial manifold expansion and semantic object injection.<n>We show that our framework significantly enhances the data efficiency and robustness of state-of-the-art models, including nnU-Net and U-Mamba.
  arXiv  Detail & Related papers  (2026-01-19T09:11:28Z)
• Forging a Dynamic Memory: Retrieval-Guided Continual Learning for Generalist Medical Foundation Models [45.285970665585914]
  We propose a comprehensive framework for Continual Learning.<n>We employ a multi-modal, multi-layer RAG system that provides real-time guidance for model fine-tuning.<n>We introduce a dynamic knowledge distillation framework.
  arXiv  Detail & Related papers  (2025-12-15T08:09:40Z)
• Med-K2N: Flexible K-to-N Modality Translation for Medical Image Synthesis [13.589690091116802]
  Cross-modal medical image synthesis research focuses on reconstructing missing imaging modalities from available ones to support clinical diagnosis.<n>How can we model the heterogeneous contributions of different modalities to various target tasks?<n>How can we maintain modality identity consistency in multi-output generation?
  arXiv  Detail & Related papers  (2025-10-03T08:47:17Z)
• Medverse: A Universal Model for Full-Resolution 3D Medical Image Segmentation, Transformation and Enhancement [15.28003304776022]
  In-context learning offers a promising paradigm for universal medical image analysis.<n>We present textbfMedverse, a universal ICL model for 3D medical imaging trained on 22 datasets.<n>Medverse employs a next-scale autoregressive in-context learning framework that progressively refines predictions from coarse to fine.
  arXiv  Detail & Related papers  (2025-09-11T08:10:49Z)
• Clinical NLP with Attention-Based Deep Learning for Multi-Disease Prediction [44.0876796031468]
  This paper addresses the challenges posed by the unstructured nature and high-dimensional semantic complexity of electronic health record texts.<n>A deep learning method based on attention mechanisms is proposed to achieve unified modeling for information extraction and multi-label disease prediction.
  arXiv  Detail & Related papers  (2025-07-02T07:45:22Z)
• Medical World Model: Generative Simulation of Tumor Evolution for Treatment Planning [50.16114813279694]
  We introduce the Medical World Model (MeWM), the first world model in medicine that visually predicts future disease states based on clinical decisions.<n>MeWM comprises (i) vision-language models to serve as policy models, and (ii) tumor generative models as dynamics models.<n>We propose the inverse dynamics model that applies survival analysis to the simulated post-treatment tumor, enabling the evaluation of treatment efficacy and the selection of the optimal clinical action plan.
  arXiv  Detail & Related papers  (2025-06-02T23:50:40Z)
• MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention [52.106879463828044]
  Histopathology and transcriptomics are fundamental modalities in oncology, encapsulating the morphological and molecular aspects of the disease. We present MIRROR, a novel multi-modal representation learning method designed to foster both modality alignment and retention. Extensive evaluations on TCGA cohorts for cancer subtyping and survival analysis highlight MIRROR's superior performance.
  arXiv  Detail & Related papers  (2025-03-01T07:02:30Z)
• SkelMamba: A State Space Model for Efficient Skeleton Action Recognition of Neurological Disorders [14.304356695180005]
  We introduce a novel state-space model (SSM)-based framework for skeleton-based human action recognition. Our model captures local joint interactions and global motion patterns across multiple body parts. This gait-aware decomposition enhances the ability to identify subtle motion patterns critical in medical diagnosis.
  arXiv  Detail & Related papers  (2024-11-29T08:43:52Z)
• Adaptive Aggregation Weights for Federated Segmentation of Pancreas MRI [5.631060921219683]
  Federated learning (FL) enables collaborative model training across institutions without sharing sensitive data. Traditional FL methods, such as Federated Averaging (FedAvg), face difficulties in generalizing across domains. This paper introduces a novel approach that incorporates adaptive aggregation weights.
  arXiv  Detail & Related papers  (2024-10-29T20:53:01Z)
• A Unified Model for Compressed Sensing MRI Across Undersampling Patterns [69.19631302047569]
  We propose a unified MRI reconstruction model robust to various measurement undersampling patterns and image resolutions. Our model improves SSIM by 11% and PSNR by 4 dB over a state-of-the-art CNN (End-to-End VarNet) with 600$times$ faster inference than diffusion methods.
  arXiv  Detail & Related papers  (2024-10-05T20:03:57Z)
• UNICORN: A Deep Learning Model for Integrating Multi-Stain Data in Histopathology [2.9389205138207277]
  UNICORN is a multi-modal transformer capable of processing multi-stain histopathology for atherosclerosis severity class prediction. The architecture comprises a two-stage, end-to-end trainable model with specialized modules utilizing transformer self-attention blocks. UNICORN achieved a classification accuracy of 0.67, outperforming other state-of-the-art models.
  arXiv  Detail & Related papers  (2024-09-26T12:13:52Z)
• Learning Brain Tumor Representation in 3D High-Resolution MR Images via Interpretable State Space Models [42.55786269051626]
  We propose a novel state-space-model (SSM)-based masked autoencoder which scales ViT-like models to handle high-resolution data effectively. We propose a latent-to-spatial mapping technique that enables direct visualization of how latent features correspond to specific regions in the input volumes. Our results highlight the potential of SSM-based self-supervised learning to transform radiomics analysis by combining efficiency and interpretability.
  arXiv  Detail & Related papers  (2024-09-12T04:36:50Z)
• Unified Modeling Enhanced Multimodal Learning for Precision Neuro-Oncology [8.802214988309684]
  We introduce a hierarchical attention structure to leverage shared and complementary features of both modalities of histology and genomics. Our method surpasses previous state-of-the-art approaches in glioma diagnosis and prognosis tasks.
  arXiv  Detail & Related papers  (2024-06-11T09:06:41Z)
• Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity [60.983327742457995]
  Reconstructing the viewed images from human brain activity bridges human and computer vision through the Brain-Computer Interface. We devise Psychometry, an omnifit model for reconstructing images from functional Magnetic Resonance Imaging (fMRI) obtained from different subjects.
  arXiv  Detail & Related papers  (2024-03-29T07:16:34Z)
• Automated ensemble method for pediatric brain tumor segmentation [0.0]
  This study introduces a novel ensemble approach using ONet and modified versions of UNet. Data augmentation ensures robustness and accuracy across different scanning protocols. Results indicate that this advanced ensemble approach offers promising prospects for enhanced diagnostic accuracy.
  arXiv  Detail & Related papers  (2023-08-14T15:29:32Z)
• Classification of lung cancer subtypes on CT images with synthetic pathological priors [41.75054301525535]
  Cross-scale associations exist in the image patterns between the same case's CT images and its pathological images. We propose self-generating hybrid feature network (SGHF-Net) for accurately classifying lung cancer subtypes on CT images.
  arXiv  Detail & Related papers  (2023-08-09T02:04:05Z)
• Ambiguous Medical Image Segmentation using Diffusion Models [60.378180265885945]
  We introduce a single diffusion model-based approach that produces multiple plausible outputs by learning a distribution over group insights. Our proposed model generates a distribution of segmentation masks by leveraging the inherent sampling process of diffusion. Comprehensive results show that our proposed approach outperforms existing state-of-the-art ambiguous segmentation networks.
  arXiv  Detail & Related papers  (2023-04-10T17:58:22Z)
• Diff-UNet: A Diffusion Embedded Network for Volumetric Segmentation [41.608617301275935]
  We propose a novel end-to-end framework, called Diff-UNet, for medical volumetric segmentation. Our approach integrates the diffusion model into a standard U-shaped architecture to extract semantic information from the input volume effectively. We evaluate our method on three datasets, including multimodal brain tumors in MRI, liver tumors, and multi-organ CT volumes.
  arXiv  Detail & Related papers  (2023-03-18T04:06:18Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)
• Adversarial Sample Enhanced Domain Adaptation: A Case Study on Predictive Modeling with Electronic Health Records [57.75125067744978]
  We propose a data augmentation method to facilitate domain adaptation. adversarially generated samples are used during domain adaptation. Results confirm the effectiveness of our method and the generality on different tasks.
  arXiv  Detail & Related papers  (2021-01-13T03:20:20Z)
• Benchmarking off-the-shelf statistical shape modeling tools in clinical applications [53.47202621511081]
  We systematically assess the outcome of widely used, state-of-the-art SSM tools. We propose validation frameworks for anatomical landmark/measurement inference and lesion screening. ShapeWorks and Deformetrica shape models are found to capture clinically relevant population-level variability.
  arXiv  Detail & Related papers  (2020-09-07T03:51:35Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.