Multimodal Federated Learning With Missing Modalities through Feature Imputation Network

• URL: http://arxiv.org/abs/2505.20232v1
• Date: Mon, 26 May 2025 17:11:03 GMT
• Title: Multimodal Federated Learning With Missing Modalities through Feature Imputation Network
• Authors: Pranav Poudel, Aavash Chhetri, Prashnna Gyawali, Georgios Leontidis, Binod Bhattarai,
• Abstract summary: Multimodal federated learning holds immense potential for collaboratively training models from multiple sources without sharing raw data.<n>Previous methods typically rely on publicly available real datasets or synthetic data to compensate for missing modalities.<n>We propose a novel, lightweight, low-dimensional feature translator to reconstruct bottleneck features of the missing modalities.
• Score: 9.384737026881504
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal federated learning holds immense potential for collaboratively training models from multiple sources without sharing raw data, addressing both data scarcity and privacy concerns, two key challenges in healthcare. A major challenge in training multimodal federated models in healthcare is the presence of missing modalities due to multiple reasons, including variations in clinical practice, cost and accessibility constraints, retrospective data collection, privacy concerns, and occasional technical or human errors. Previous methods typically rely on publicly available real datasets or synthetic data to compensate for missing modalities. However, obtaining real datasets for every disease is impractical, and training generative models to synthesize missing modalities is computationally expensive and prone to errors due to the high dimensionality of medical data. In this paper, we propose a novel, lightweight, low-dimensional feature translator to reconstruct bottleneck features of the missing modalities. Our experiments on three different datasets (MIMIC-CXR, NIH Open-I, and CheXpert), in both homogeneous and heterogeneous settings consistently improve the performance of competitive baselines. The code and implementation details are available at: https://github.com/bhattarailab/FedFeatGen

Related papers

• Continual Multimodal Contrastive Learning [70.60542106731813]
  Multimodal contrastive learning (MCL) advances in aligning different modalities and generating multimodal representations in a joint space.<n>However, a critical yet often overlooked challenge remains: multimodal data is rarely collected in a single process, and training from scratch is computationally expensive.<n>In this paper, we formulate CMCL through two specialized principles of stability and plasticity.<n>We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient is prevented from interfering with the previously learned knowledge.
  arXiv  Detail & Related papers  (2025-03-19T07:57:08Z)
• Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment [0.8213829427624407]
  We propose a new multi-model model called MoSARe, a deep learning framework that handles incomplete multimodal data.<n>MoSARe integrates expert selection, cross-modal attention, and contrastive learning to improve feature representation and decision-making.<n>It provides reliable predictions even when some data are missing.
  arXiv  Detail & Related papers  (2025-03-12T16:03:00Z)
• Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification [2.5091334993691206]
  Development of a robust deep-learning model for retinal disease diagnosis requires a substantial dataset for training. The capacity to generalize effectively on smaller datasets remains a persistent challenge. We've combined a wide range of data sources to improve performance and generalization to new data.
  arXiv  Detail & Related papers  (2024-09-17T17:22:35Z)
• CAR-MFL: Cross-Modal Augmentation by Retrieval for Multimodal Federated Learning with Missing Modalities [6.336606641921228]
  We propose a novel method for multimodal federated learning with missing modalities. Our contribution lies in a novel cross-modal data augmentation by retrieval, leveraging the small publicly available dataset. Our method learns the parameters in a federated manner, ensuring privacy protection and improving performance.
  arXiv  Detail & Related papers  (2024-07-11T16:26:08Z)
• Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos [92.38662956154256]
  Real-world applications often face challenges with incomplete modalities due to privacy concerns, efficiency needs, or hardware issues.<n>We propose a novel approach to address this issue at test time without requiring retraining.<n>MiDl represents the first self-supervised, online solution for handling missing modalities exclusively at test time.
  arXiv  Detail & Related papers  (2024-04-23T16:01:33Z)
• FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology [3.802258033231335]
  Federated Multi-Modal (FedMM) is a learning framework that trains multiple single-modal feature extractors to enhance subsequent classification performance. FedMM notably outperforms two baselines in accuracy and AUC metrics.
  arXiv  Detail & Related papers  (2024-02-24T16:58:42Z)
• UnitedHuman: Harnessing Multi-Source Data for High-Resolution Human Generation [59.77275587857252]
  A holistic human dataset inevitably has insufficient and low-resolution information on local parts. We propose to use multi-source datasets with various resolution images to jointly learn a high-resolution human generative model.
  arXiv  Detail & Related papers  (2023-09-25T17:58:46Z)
• Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data [8.536869574065195]
  Multi-Modal Mixing Transformer (3MAT) is a disease classification transformer that not only leverages multi-modal data but also handles missing data scenarios. We propose a novel modality dropout mechanism to ensure an unprecedented level of modality independence and robustness to handle missing data scenarios.
  arXiv  Detail & Related papers  (2022-10-01T11:31:02Z)
• Practical Challenges in Differentially-Private Federated Survival Analysis of Medical Data [57.19441629270029]
  In this paper, we take advantage of the inherent properties of neural networks to federate the process of training of survival analysis models. In the realistic setting of small medical datasets and only a few data centers, this noise makes it harder for the models to converge. We propose DPFed-post which adds a post-processing stage to the private federated learning scheme.
  arXiv  Detail & Related papers  (2022-02-08T10:03:24Z)
• FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesis [55.939957482776194]
  We propose a new benchmark for federated domain translation on unsupervised brain image synthesis (termed as FedMed-GAN) FedMed-GAN mitigates the mode collapse without sacrificing the performance of generators. A comprehensive evaluation is provided for comparing FedMed-GAN and other centralized methods.
  arXiv  Detail & Related papers  (2022-01-22T02:50:29Z)
• Brain Image Synthesis with Unsupervised Multivariate Canonical CSC$\ell_4$Net [122.8907826672382]
  We propose to learn dedicated features that cross both intre- and intra-modal variations using a novel CSC$ell_4$Net.
  arXiv  Detail & Related papers  (2021-03-22T05:19:40Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.