Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architectures

• URL: http://arxiv.org/abs/2509.03695v1
• Date: Wed, 03 Sep 2025 20:23:19 GMT
• Title: Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architectures
• Authors: Payam Abdisarabshali, Fardis Nadimi, Kasra Borazjani, Naji Khosravan, Minghui Liwang, Wei Ni, Dusit Niyato, Michael Langberg, Seyyedali Hosseinalipour,
• Abstract summary: In this paper, we unveil an unexplored variation of M3T FFMs by proposing hierarchical federated foundation models (HF-FMs)<n>HF-FMs strategically align the modular structure of M3T FMs, comprising modality encoders, prompts, mixture-of-experts (MoEs), adapters, and task heads.<n>To demonstrate their potential, we prototype HF-FMs in a wireless network setting and release the open-source code for the development of HF-FMs.
• Score: 58.72593025539547
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rise of foundation models (FMs) has reshaped the landscape of machine learning. As these models continued to grow, leveraging geo-distributed data from wireless devices has become increasingly critical, giving rise to federated foundation models (FFMs). More recently, FMs have evolved into multi-modal multi-task (M3T) FMs (e.g., GPT-4) capable of processing diverse modalities across multiple tasks, which motivates a new underexplored paradigm: M3T FFMs. In this paper, we unveil an unexplored variation of M3T FFMs by proposing hierarchical federated foundation models (HF-FMs), which in turn expose two overlooked heterogeneity dimensions to fog/edge networks that have a direct impact on these emerging models: (i) heterogeneity in collected modalities and (ii) heterogeneity in executed tasks across fog/edge nodes. HF-FMs strategically align the modular structure of M3T FMs, comprising modality encoders, prompts, mixture-of-experts (MoEs), adapters, and task heads, with the hierarchical nature of fog/edge infrastructures. Moreover, HF-FMs enable the optional usage of device-to-device (D2D) communications, enabling horizontal module relaying and localized cooperative training among nodes when feasible. Through delving into the architectural design of HF-FMs, we highlight their unique capabilities along with a series of tailored future research directions. Finally, to demonstrate their potential, we prototype HF-FMs in a wireless network setting and release the open-source code for the development of HF-FMs with the goal of fostering exploration in this untapped field (GitHub: https://github.com/payamsiabd/M3T-FFM).

Related papers

• Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Survey [9.762879334040566]
  Foundation models (FMs) are recognized as a transformative breakthrough that has started to reshape the future of artificial intelligence (AI)<n>This work discusses the utilization of FMs, especially multi-modal FMs in wireless networks.
  arXiv  Detail & Related papers  (2026-01-06T16:59:29Z)
• Multimodal Wireless Foundation Models [7.397099215417549]
  We build the first multimodal wireless foundation model capable of processing both raw IQ streams and image-like wireless modalities.<n>We evaluate the model on five tasks across both modality families: image-based (human activity sensing, RF signal classification, 5G NR positioning) and IQ-based (RF device fingerprinting, interference detection/classification)
  arXiv  Detail & Related papers  (2025-11-19T06:26:49Z)
• Synergies between Federated Foundation Models and Smart Power Grids [8.179321682277818]
  M3T Federated Foundation Models (FedFMs) enable scalable, privacy-preserving model training/fine-tuning across distributed data sources.<n>In this paper, we take one of the first steps toward introducing these models to the power systems research community.
  arXiv  Detail & Related papers  (2025-09-20T02:00:07Z)
• Specialized Foundation Models Struggle to Beat Supervised Baselines [60.23386520331143]
  We look at three modalities -- genomics, satellite imaging, and time series -- with multiple recent FMs and compare them to a standard supervised learning workflow.<n>We find that it is consistently possible to train simple supervised models that match or even outperform the latest foundation models.
  arXiv  Detail & Related papers  (2024-11-05T04:10:59Z)
• FedPFT: Federated Proxy Fine-Tuning of Foundation Models [55.58899993272904]
  Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges as a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. We propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules.
  arXiv  Detail & Related papers  (2024-04-17T16:30:06Z)
• Forging Vision Foundation Models for Autonomous Driving: Challenges, Methodologies, and Opportunities [59.02391344178202]
  Vision foundation models (VFMs) serve as potent building blocks for a wide range of AI applications. The scarcity of comprehensive training data, the need for multi-sensor integration, and the diverse task-specific architectures pose significant obstacles to the development of VFMs. This paper delves into the critical challenge of forging VFMs tailored specifically for autonomous driving, while also outlining future directions.
  arXiv  Detail & Related papers  (2024-01-16T01:57:24Z)
• EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge [15.559604113977294]
  We propose EdgeFM, a novel edge-cloud cooperative system with open-set recognition capability. We show that EdgeFM can reduce the end-to-end latency up to 3.2x and achieve 34.3% accuracy increase compared with the baseline.
  arXiv  Detail & Related papers  (2023-11-18T06:40:39Z)
• The Role of Federated Learning in a Wireless World with Foundation Models [59.8129893837421]
  Foundation models (FMs) are general-purpose artificial intelligence (AI) models that have recently enabled multiple brand-new generative AI applications. Currently, the exploration of the interplay between FMs and federated learning (FL) is still in its nascent stage. This article explores the extent to which FMs are suitable for FL over wireless networks, including a broad overview of research challenges and opportunities.
  arXiv  Detail & Related papers  (2023-10-06T04:13:10Z)
• When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions [57.91211653592199]
  The intersection of Foundation Model (FM) and Federated Learning (FL) presents a unique opportunity to unlock new possibilities for real-world applications.<n>On the one hand, FL, as a collaborative learning paradigm, help address challenges in FM development by expanding data availability.<n>On the other hand, FM, equipped with pre-trained knowledge and exceptional performance, can serve as a robust starting point for FL.
  arXiv  Detail & Related papers  (2023-06-27T15:15:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.