Enhancing Robustness to Missing Modalities through Clustered Federated Learning

• URL: http://arxiv.org/abs/2505.06911v2
• Date: Wed, 02 Jul 2025 13:11:28 GMT
• Title: Enhancing Robustness to Missing Modalities through Clustered Federated Learning
• Authors: Lishan Yang, Wei Emma Zhang, Quan Z. Sheng, Weitong Chen, Lina Yao, Weitong Chen, Ali Shakeri,
• Abstract summary: We present MMiC, a framework for Mitigating Modality incompleteness in Multimodal Federated Learning.<n> MMiC replaces partial parameters within client models inside clusters to mitigate the impact of missing modalities.<n> MMiC consistently outperforms existing federated learning architectures in both global and personalized performance.
• Score: 29.945585700688373
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In the era of big data, data mining has become indispensable for uncovering hidden patterns and insights from vast and complex datasets. The integration of multimodal data sources further enhances its potential. Multimodal Federated Learning (MFL) is a distributed approach that enhances the efficiency and quality of multimodal learning, ensuring collaborative work and privacy protection. However, missing modalities pose a significant challenge in MFL, often due to data quality issues or privacy policies across the clients. In this work, we present MMiC, a framework for Mitigating Modality incompleteness in MFL within the Clusters. MMiC replaces partial parameters within client models inside clusters to mitigate the impact of missing modalities. Furthermore, it leverages the Banzhaf Power Index to optimize client selection under these conditions. Finally, MMiC employs an innovative approach to dynamically control global aggregation by utilizing Markovitz Portfolio Optimization. Extensive experiments demonstrate that MMiC consistently outperforms existing federated learning architectures in both global and personalized performance on multimodal datasets with missing modalities, confirming the effectiveness of our proposed solution.

Related papers

• Not All Clients Are Equal: Personalized Federated Learning on Heterogeneous Multi-Modal Clients [52.14230635007546]
  Foundation models have shown remarkable capabilities across diverse multi-modal tasks, but their centralized training raises privacy concerns and induces high transmission costs.<n>For the growing demand for personalizing AI models for different user purposes, personalized federated learning (PFL) has emerged.<n>PFL allows each client to leverage the knowledge of other clients for further adaptation to individual user preferences, again without the need to share data.
  arXiv  Detail & Related papers  (2025-05-20T09:17:07Z)
• Client-Centric Federated Adaptive Optimization [78.30827455292827]
  Federated Learning (FL) is a distributed learning paradigm where clients collaboratively train a model while keeping their own data private.<n>We propose Federated-Centric Adaptive Optimization, which is a class of novel federated optimization approaches.
  arXiv  Detail & Related papers  (2025-01-17T04:00:50Z)
• MetaTrading: An Immersion-Aware Model Trading Framework for Vehicular Metaverse Services [94.61039892220037]
  We propose an immersion-aware model trading framework that facilitates data provision for services while ensuring privacy through federated learning (FL)<n>We design an incentive mechanism to incentivize metaverse users (MUs) to contribute high-value models under resource constraints.<n>We develop a fully distributed dynamic reward algorithm based on deep reinforcement learning, without accessing any private information about MUs and other MSPs.
  arXiv  Detail & Related papers  (2024-10-25T16:20:46Z)
• Adversarial Federated Consensus Learning for Surface Defect Classification Under Data Heterogeneity in IIoT [8.48069043458347]
  It's difficult to collect and centralize sufficient training data from various entities in Industrial Internet of Things (IIoT) Federated learning (FL) provides a solution by enabling collaborative global model training across clients. We propose a novel personalized FL approach, named Adversarial Federated Consensus Learning (AFedCL)
  arXiv  Detail & Related papers  (2024-09-24T03:59:32Z)
• FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization [11.040916982022978]
  Federated Learning (FL) enables collaborative training of machine learning models on decentralized data. Data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena. We propose a novel Bayesian PFL framework using bi-level optimization to tackle the data heterogeneity challenges.
  arXiv  Detail & Related papers  (2024-05-29T11:28:06Z)
• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality [28.90486547668949]
  Multimodal Federated Cross Prototype Learning (MFCPL) is a novel approach for MFL under severely missing modalities.<n>MFCPL provides diverse modality knowledge in modality-shared level with the cross-modal regularization and modality-specific level with cross-modal contrastive mechanism.<n>Our approach introduces the cross-modal alignment to provide regularization for modality-specific features, thereby enhancing the overall performance.
  arXiv  Detail & Related papers  (2024-01-25T02:25:23Z)
• Balanced Multi-modal Federated Learning via Cross-Modal Infiltration [19.513099949266156]
  Federated learning (FL) underpins advancements in privacy-preserving distributed computing. We propose a novel Cross-Modal Infiltration Federated Learning (FedCMI) framework.
  arXiv  Detail & Related papers  (2023-12-31T05:50:15Z)
• Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection [19.284989473603627]
  We propose a novel Balanced Modality Selection framework for multi-modal learning (MFL) We show that local training with a certain single modality may contribute more to the global model than training with all local modalities. Our experiments on audio-visual, colored-gray, and front-back datasets showcase the superiority of BMSFed over baselines.
  arXiv  Detail & Related papers  (2023-12-31T05:37:27Z)
• FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication [11.254610576923204]
  We propose Federated Multimodal Fusion learning with Selective modality communication (FedMFS) Key idea is the introduction of a modality selection criterion for each device, which weighs (i) the impact of the modality, gauged by Shapley value analysis, against (ii) the modality model size as a gauge for communication overhead. Experiments on the real-world ActionSense dataset demonstrate the ability of FedMFS to achieve comparable accuracy to several baselines while reducing the communication overhead by over 4x.
  arXiv  Detail & Related papers  (2023-10-10T22:23:27Z)
• Personalized Federated Learning under Mixture of Distributions [98.25444470990107]
  We propose a novel approach to Personalized Federated Learning (PFL), which utilizes Gaussian mixture models (GMM) to fit the input data distributions across diverse clients. FedGMM possesses an additional advantage of adapting to new clients with minimal overhead, and it also enables uncertainty quantification. Empirical evaluations on synthetic and benchmark datasets demonstrate the superior performance of our method in both PFL classification and novel sample detection.
  arXiv  Detail & Related papers  (2023-05-01T20:04:46Z)
• Multimodal Federated Learning via Contrastive Representation Ensemble [17.08211358391482]
  Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. Existing FL methods all rely on model aggregation on single modality level. We propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL)
  arXiv  Detail & Related papers  (2023-02-17T14:17:44Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.