FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in
Computational Pathology
- URL: http://arxiv.org/abs/2402.15858v1
- Date: Sat, 24 Feb 2024 16:58:42 GMT
- Title: FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in
Computational Pathology
- Authors: Yuanzhe Peng, Jieming Bian, Jie Xu
- Abstract summary: Federated Multi-Modal (FedMM) is a learning framework that trains multiple single-modal feature extractors to enhance subsequent classification performance.
FedMM notably outperforms two baselines in accuracy and AUC metrics.
- Score: 3.802258033231335
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The fusion of complementary multimodal information is crucial in
computational pathology for accurate diagnostics. However, existing multimodal
learning approaches necessitate access to users' raw data, posing substantial
privacy risks. While Federated Learning (FL) serves as a privacy-preserving
alternative, it falls short in addressing the challenges posed by heterogeneous
(yet possibly overlapped) modalities data across various hospitals. To bridge
this gap, we propose a Federated Multi-Modal (FedMM) learning framework that
federatedly trains multiple single-modal feature extractors to enhance
subsequent classification performance instead of existing FL that aims to train
a unified multimodal fusion model. Any participating hospital, even with
small-scale datasets or limited devices, can leverage these federated trained
extractors to perform local downstream tasks (e.g., classification) while
ensuring data privacy. Through comprehensive evaluations of two publicly
available datasets, we demonstrate that FedMM notably outperforms two baselines
in accuracy and AUC metrics.
Related papers
- CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities [6.336606641921228]
We propose a novel method for multimodal federated learning with missing modalities.
Our contribution lies in a novel cross-modal data augmentation by retrieval, leveraging the small publicly available dataset.
Our method learns the parameters in a federated manner, ensuring privacy protection and improving performance.
arXiv Detail & Related papers (2024-07-11T16:26:08Z) - Completed Feature Disentanglement Learning for Multimodal MRIs Analysis [36.32164729310868]
Feature disentanglement (FD)-based methods have achieved significant success in multimodal learning (MML)
We propose a novel Complete Feature Disentanglement (CFD) strategy that recovers the lost information during feature decoupling.
Specifically, the CFD strategy not only identifies modality-shared and modality-specific features, but also decouples shared features among subsets of multimodal inputs.
arXiv Detail & Related papers (2024-07-06T01:49:38Z) - Multi-Modal Federated Learning for Cancer Staging over Non-IID Datasets with Unbalanced Modalities [9.476402318365446]
In this work, we introduce a novel FL architecture designed to accommodate not only the heterogeneity of data samples, but also the inherent heterogeneity/non-uniformity of data modalities across institutions.
We propose a solution by devising a distributed gradient blending and proximity-aware client weighting strategy tailored for multi-modal FL.
arXiv Detail & Related papers (2024-01-07T23:45:01Z) - XAI for In-hospital Mortality Prediction via Multimodal ICU Data [57.73357047856416]
We propose an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data.
We employ multimodal learning in our framework, which can receive heterogeneous inputs from clinical data and make decisions.
Our framework can be easily transferred to other clinical tasks, which facilitates the discovery of crucial factors in healthcare research.
arXiv Detail & Related papers (2023-12-29T14:28:04Z) - Source-Free Collaborative Domain Adaptation via Multi-Perspective
Feature Enrichment for Functional MRI Analysis [55.03872260158717]
Resting-state MRI functional (rs-fMRI) is increasingly employed in multi-site research to aid neurological disorder analysis.
Many methods have been proposed to reduce fMRI heterogeneity between source and target domains.
But acquiring source data is challenging due to concerns and/or data storage burdens in multi-site studies.
We design a source-free collaborative domain adaptation framework for fMRI analysis, where only a pretrained source model and unlabeled target data are accessible.
arXiv Detail & Related papers (2023-08-24T01:30:18Z) - Cross-Modal Vertical Federated Learning for MRI Reconstruction [42.527873703840996]
Federated learning enables multiple hospitals to cooperatively learn a shared model without privacy disclosure.
We develop a novel framework, namely Federated Consistent Regularization constrained Feature Disentanglement (Fed-CRFD), for boosting MRI reconstruction.
Our method can fully exploit the multi-source data from hospitals while alleviating the domain shift problem.
arXiv Detail & Related papers (2023-06-05T08:07:01Z) - Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning
with Hierarchical Aggregation [16.308470947384134]
HA-Fedformer is a novel transformer-based model that empowers unimodal training with only a unimodal dataset at the client.
We develop an uncertainty-aware aggregation method for the local encoders with layer-wise Markov Chain Monte Carlo sampling.
Our experiments on popular sentiment analysis benchmarks, CMU-MOSI and CMU-MOSEI, demonstrate that HA-Fedformer significantly outperforms state-of-the-art multimodal models.
arXiv Detail & Related papers (2023-03-27T07:07:33Z) - Federated Cycling (FedCy): Semi-supervised Federated Learning of
Surgical Phases [57.90226879210227]
FedCy is a semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos.
We demonstrate significant performance gains over state-of-the-art FSSL methods on the task of automatic recognition of surgical phases.
arXiv Detail & Related papers (2022-03-14T17:44:53Z) - Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal
Sentiment Analysis [96.46952672172021]
Bi-Bimodal Fusion Network (BBFN) is a novel end-to-end network that performs fusion on pairwise modality representations.
Model takes two bimodal pairs as input due to known information imbalance among modalities.
arXiv Detail & Related papers (2021-07-28T23:33:42Z) - Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without
Sharing Private Information [55.866673486753115]
We propose an extendable and elastic learning framework to preserve privacy and security.
The proposed framework is named distributed Asynchronized Discriminator Generative Adrial Networks (AsynDGAN)
arXiv Detail & Related papers (2020-12-15T20:41:24Z) - MS-Net: Multi-Site Network for Improving Prostate Segmentation with
Heterogeneous MRI Data [75.73881040581767]
We propose a novel multi-site network (MS-Net) for improving prostate segmentation by learning robust representations.
Our MS-Net improves the performance across all datasets consistently, and outperforms state-of-the-art methods for multi-site learning.
arXiv Detail & Related papers (2020-02-09T14:11:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.