FedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities

• URL: http://arxiv.org/abs/2607.23245v1
• Date: Sat, 25 Jul 2026 15:11:58 GMT
• Title: FedTaste: Topology-Aware Structural Transfer for Multimodal Federated Learning with Missing Modalities
• Abstract summary: FedTaste is a parameter-efficient framework for topology-aware structural transfer in Multimodal Federated Learning.<n>To adapt clients with missing modalities, we introduce Modality-Adaptive Structural Prompts.<n>In this way, FedTaste avoids explicit modality imputation while preserving shared semantic structure across clients.
• Score: 7.015345209610299
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Federated Learning is often challenged by arbitrary modality missingness and Non-IID data distributions, which lead to severe representation drift and hinder effective collaboration across clients. Existing methods typically rely on generative imputation, external auxiliary data, or isolated unimodal training to bridge modality gaps, often incurring substantial communication and computational costs as well as potential privacy risks. To address these limitations, we propose FedTaste, a parameter-efficient framework for topology-aware structural transfer in Multimodal Federated Learning with missing modalities. Instead of aligning fragile first-order features, FedTaste focuses on more stable group-level semantic relations. Specifically, FedTaste leverages frozen foundation models to extract a joint multimodal topology from full-modality clients, which is then consolidated by the server into a global structural blueprint. To adapt clients with missing modalities, we introduce Modality-Adaptive Structural Prompts together with spectral consistency regularization, enabling lightweight branch-specific adaptation that aligns local partial representations with the shared blueprint. In this way, FedTaste avoids explicit modality imputation while preserving shared semantic structure across clients. Extensive experiments demonstrate that FedTaste consistently achieves superior performance across multiple datasets and challenging Non-IID settings, while substantially reducing communication overhead compared with existing methods.

Related papers

• Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients [22.724544289853558]
  FedSLM is a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients.<n>We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise.
  arXiv  Detail & Related papers  (2026-07-31T06:45:56Z)
• Boosting Multimodal Federated Learning via Chained Modality Optimization [69.53075328069504]
  Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability.<n>To address this, we propose FedMChain, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases.<n>This phase-wise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an error-compensated regularizer.
  arXiv  Detail & Related papers  (2026-06-01T08:07:09Z)
• Towards Robust Federated Multimodal Graph Learning under Modality Heterogeneity [49.44748281840733]
  Real-world graphs are often isolated due to data-sharing limitations across multiple parties.<n>We propose scFedMPO, which utilizes topology-aware cross-modal generation to recover missing features.<n>Experiments on 3 tasks across 6 datasets demonstrate that FedMPO outperforms baselines.
  arXiv  Detail & Related papers  (2026-05-12T17:30:15Z)
• PLUME: Latent Reasoning Based Universal Multimodal Embedding [52.35354073629127]
  Universal multimodal embedding (UME) maps heterogeneous inputs into a shared retrieval space with a single model.<n>Recent approaches improve UME by generating explicit chain-of-thought (CoT) rationales before extracting embeddings.<n>We propose PLUME, a latent reasoning framework that advances UME by replacing verbalized CoT with a short autoregressive rollout of continuous latent states.
  arXiv  Detail & Related papers  (2026-04-02T14:04:53Z)
• Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity [27.791732281031695]
  Large language models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale model capacity while reducing computation.<n>Fine-tuning these MoE-based LLMs often requires access to distributed and privacy-sensitive data, making centralized fine-tuning impractical.<n>We propose FedAlign-MoE, a federated aggregation alignment framework that jointly enforces routing consistency and expert semantic alignment.
  arXiv  Detail & Related papers  (2026-03-22T15:07:39Z)
• Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data [22.933465523798475]
  This paper bridges the gap between federated learning and multi-modal prompt-tuning.<n>A key challenge in this setting arises from the lack of semantic alignment between prompt instructions.<n>Our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate prompt-tuning instructions.
  arXiv  Detail & Related papers  (2026-02-06T03:53:35Z)
• Federated Multi-Task Clustering [44.73672172790804]
  This paper proposes a novel framework named Federated Multi-Task Clustering (i.e.,FMTC)<n>It is composed of two main components: client-side personalized clustering module and server-side tensorial correlation module.<n>We derive an efficient, privacy-preserving distributed algorithm based on the Alternating Direction Method of Multipliers.
  arXiv  Detail & Related papers  (2025-12-28T12:02:32Z)
• Towards Federated Clustering: A Client-wise Private Graph Aggregation Framework [57.04850867402913]
  Federated clustering addresses the challenge of extracting patterns from decentralized, unlabeled data.<n>We propose Structural Privacy-Preserving Federated Graph Clustering (SPP-FGC), a novel algorithm that innovatively leverages local structural graphs as the primary medium for privacy-preserving knowledge sharing.<n>Our framework achieves state-of-the-art performance, improving clustering accuracy by up to 10% (NMI) over federated baselines while maintaining provable privacy guarantees.
  arXiv  Detail & Related papers  (2025-11-14T03:05:22Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models [23.830133838392964]
  We propose FedRSCLIP, the first federated learning framework for remote sensing image classification based on a VLM, specifically CLIP.<n>FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning.<n>To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets.
  arXiv  Detail & Related papers  (2025-01-05T07:10:27Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.