FedMultimodal: A Benchmark For Multimodal Federated Learning

• URL: http://arxiv.org/abs/2306.09486v2
• Date: Tue, 20 Jun 2023 20:26:36 GMT
• Title: FedMultimodal: A Benchmark For Multimodal Federated Learning
• Authors: Tiantian Feng and Digbalay Bose and Tuo Zhang and Rajat Hebbar and Anil Ramakrishna and Rahul Gupta and Mi Zhang and Salman Avestimehr and Shrikanth Narayanan
• Abstract summary: Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges. Despite significant efforts to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. We introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets.
• Score: 45.0258566979478
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Over the past few years, Federated Learning (FL) has become an emerging machine learning technique to tackle data privacy challenges through collaborative training. In the Federated Learning algorithm, the clients submit a locally trained model, and the server aggregates these parameters until convergence. Despite significant efforts that have been made to FL in fields like computer vision, audio, and natural language processing, the FL applications utilizing multimodal data streams remain largely unexplored. It is known that multimodal learning has broad real-world applications in emotion recognition, healthcare, multimedia, and social media, while user privacy persists as a critical concern. Specifically, there are no existing FL benchmarks targeting multimodal applications or related tasks. In order to facilitate the research in multimodal FL, we introduce FedMultimodal, the first FL benchmark for multimodal learning covering five representative multimodal applications from ten commonly used datasets with a total of eight unique modalities. FedMultimodal offers a systematic FL pipeline, enabling end-to-end modeling framework ranging from data partition and feature extraction to FL benchmark algorithms and model evaluation. Unlike existing FL benchmarks, FedMultimodal provides a standardized approach to assess the robustness of FL against three common data corruptions in real-life multimodal applications: missing modalities, missing labels, and erroneous labels. We hope that FedMultimodal can accelerate numerous future research directions, including designing multimodal FL algorithms toward extreme data heterogeneity, robustness multimodal FL, and efficient multimodal FL. The datasets and benchmark results can be accessed at: https://github.com/usc-sail/fed-multimodal.

Related papers

• FedEPA: Enhancing Personalization and Modality Alignment in Multimodal Federated Learning [9.531634844824596]
  Federated Learning (FL) enables decentralized model training across multiple parties while preserving privacy. Most FL systems assume clients hold only unimodal data, limiting their real-world applicability. We propose FedEPA, a novel FL framework for multimodal learning.
  arXiv  Detail & Related papers  (2025-04-16T12:32:37Z)
• FedMobile: Enabling Knowledge Contribution-aware Multi-modal Federated Learning with Incomplete Modalities [15.771749047384535]
  Key challenge in mobile sensing systems using multimodal FL is modality incompleteness. Current multimodal FL frameworks typically train multiple unimodal FL subsystems or apply techniques on the node side to approximate missing modalities. We present FedMobile, a new knowledge contribution-aware multimodal FL framework designed for robust learning despite missing modalities.
  arXiv  Detail & Related papers  (2025-02-20T15:10:43Z)
• FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data [64.50893177169996]
  Fine-tuning Multimodal Large Language Models (MLLMs) with Federated Learning (FL) allows for expanding the training data scope by including private data sources. We introduce a benchmark for evaluating various downstream tasks in the federated fine-tuning of MLLMs within multimodal heterogeneous scenarios. We develop a general FedMLLM framework that integrates four representative FL methods alongside two modality-agnostic strategies.
  arXiv  Detail & Related papers  (2024-11-22T04:09:23Z)
• FedModule: A Modular Federated Learning Framework [5.872098693249397]
  Federated learning (FL) has been widely adopted across various applications, such as healthcare, finance, and smart cities. This paper introduces FedModule, a flexible and FL experimental framework. FedModule adheres to the "one code, all scenarios" principle and employs a modular design that breaks the FL process into individual components.
  arXiv  Detail & Related papers  (2024-09-07T15:03:12Z)
• FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learning [42.84295556492631]
  Federated learning (FL) enables a decentralized machine learning paradigm for multiple clients to collaboratively train a generalized global model without sharing their private data. Most existing works simply propose typical FL systems for single-modal data, thus limiting its potential on exploiting valuable multimodal data for future personalized applications. We propose a novel multimodal FL framework that employs a semi-supervised learning approach to leverage the representations from different modalities. Bringing this concept into a system, we develop a distillation-based multimodal embedding knowledge transfer mechanism, namely FedMEKT, which allows the server and clients to exchange the joint knowledge of their learning models extracted from a small multimodal
  arXiv  Detail & Related papers  (2023-07-25T02:55:33Z)
• FedWon: Triumphing Multi-domain Federated Learning Without Normalization [50.49210227068574]
  Federated learning (FL) enhances data privacy with collaborative in-situ training on decentralized clients. However, Federated learning (FL) encounters challenges due to non-independent and identically distributed (non-i.i.d) data. We propose a novel method called Federated learning Without normalizations (FedWon) to address the multi-domain problem in FL.
  arXiv  Detail & Related papers  (2023-06-09T13:18:50Z)
• Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation [16.308470947384134]
  HA-Fedformer is a novel transformer-based model that empowers unimodal training with only a unimodal dataset at the client. We develop an uncertainty-aware aggregation method for the local encoders with layer-wise Markov Chain Monte Carlo sampling. Our experiments on popular sentiment analysis benchmarks, CMU-MOSI and CMU-MOSEI, demonstrate that HA-Fedformer significantly outperforms state-of-the-art multimodal models.
  arXiv  Detail & Related papers  (2023-03-27T07:07:33Z)
• FS-Real: Towards Real-World Cross-Device Federated Learning [60.91678132132229]
  Federated Learning (FL) aims to train high-quality models in collaboration with distributed clients while not uploading their local data. There is still a considerable gap between the flourishing FL research and real-world scenarios, mainly caused by the characteristics of heterogeneous devices and its scales. We propose an efficient and scalable prototyping system for real-world cross-device FL, FS-Real.
  arXiv  Detail & Related papers  (2023-03-23T15:37:17Z)
• Multimodal Federated Learning via Contrastive Representation Ensemble [17.08211358391482]
  Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. Existing FL methods all rely on model aggregation on single modality level. We propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL)
  arXiv  Detail & Related papers  (2023-02-17T14:17:44Z)
• FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings [51.09574369310246]
  Federated Learning (FL) is a novel approach enabling several clients holding sensitive data to collaboratively train machine learning models. We propose a novel cross-silo dataset suite focused on healthcare, FLamby, to bridge the gap between theory and practice of cross-silo FL. Our flexible and modular suite allows researchers to easily download datasets, reproduce results and re-use the different components for their research.
  arXiv  Detail & Related papers  (2022-10-10T12:17:30Z)
• Multi-Center Federated Learning [62.32725938999433]
  Federated learning (FL) can protect data privacy in distributed learning. It merely collects local gradients from users without access to their data. We propose a novel multi-center aggregation mechanism.
  arXiv  Detail & Related papers  (2021-08-19T12:20:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.