FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA

• URL: http://arxiv.org/abs/2607.26618v1
• Date: Wed, 29 Jul 2026 08:47:58 GMT
• Title: FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA
• Abstract summary: Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples.<n>Existing methods typically specialize at client granularity, implicitly assuming task-coherent clients.<n>We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization.
• Score: 15.068543696107227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples. However, task heterogeneity across clients can cause cross-task interference and gradient conflicts during aggregation. Federated MoE-LoRA addresses this challenge through specialized LoRA experts and conditional routing. Yet existing methods typically specialize at client granularity, implicitly assuming task-coherent clients. Our core insight is that experts need purity, namely pattern-coherent updates that preserve specialization, whereas routers need contrast, namely mixed-task observations that support expert comparison. We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization to meet these two requirements. FedWeave uses unsupervised prototype discovery to form local buckets and align them across clients, enabling prototype-level expert aggregation while retaining mixed-task client trajectories for router training. At inference, FedWeave performs sparse inference with one active expert while preserving nearly all soft-routing performance. Our theoretical analysis explains why asymmetric aggregation is advantageous: it controls expert convergence in stationarity through off-pattern contamination, identifies the consensus error induced by fragmented router trajectories, and bounds sparse-inference risk. On a heterogeneous multi-task benchmark with mainstream LLM backbones, FedWeave consistently outperforms strong baselines, while ablations verify the effectiveness of our design.

Related papers

• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• OrderMoE: An expert similarity driven distributed edge MoE inference [46.66948323823587]
  This paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE.<n>It aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality.<n> Experimental results show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio.
  arXiv  Detail & Related papers  (2026-07-19T09:20:21Z)
• FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts [0.0]
  We introduce Fractional-Fourier Mixture of Experts, a mixture-of-experts adapter.<n>Every expert carries a learnable fractional-Fourier order that continuously interpolates between the spatial domain and the Fourier domain.<n>The order is a single scalar per expert, trained with a separate chirp, and the transform is computed with an $mathcalOdlog d)$--FFT surrogate.
  arXiv  Detail & Related papers  (2026-06-30T20:39:55Z)
• ResMerge: Residual-based Spectral Merging of Large Language Models [52.65011413018957]
  We propose ResMerge, a residual-based spectral merging framework for RL experts.<n>Based on this observation, we propose ResMerge, a residual-based spectral merging framework for RL experts.
  arXiv  Detail & Related papers  (2026-06-01T13:42:45Z)
• Multi-Domain Learning with Global Expert Mapping [102.62297074508147]
  Mixture-of-Experts (MoE) models provide a scalable solution by routing inputs to specializedworks (experts)<n>We propose GEM (Global Expert Mapping), a planner-compiler framework that replaces the learned router with a global scheduler.<n>Our planner, based on linear programming relaxation, computes a fractional assignment of datasets to experts, while the compiler applies hierarchical rounding to convert this soft plan into a deterministic, capacity-aware mapping.<n>Experiments show that GEM-DINO achieves state-of-the-art performance on the UODB benchmark, with notable gains on underrepresented datasets and solves task interference in few
  arXiv  Detail & Related papers  (2026-04-20T21:09:34Z)
• Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity [27.791732281031695]
  Large language models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale model capacity while reducing computation.<n>Fine-tuning these MoE-based LLMs often requires access to distributed and privacy-sensitive data, making centralized fine-tuning impractical.<n>We propose FedAlign-MoE, a federated aggregation alignment framework that jointly enforces routing consistency and expert semantic alignment.
  arXiv  Detail & Related papers  (2026-03-22T15:07:39Z)
• Phase-Aware Mixture of Experts for Agentic Reinforcement Learning [23.18318273534301]
  A plausible remedy could be employing the Mixture-of-Experts (MoE) architecture in the policy network.<n>MoE allows different parameters (experts) to specialize in different tasks, preventing simple tasks from dominating all parameters.<n>We propose textbfPhase-Aware Mixture of Experts (PA-MoE).<n>It first features a lightweight emphphase router that learns latent phase boundaries directly from the RL objective without pre-defining phase categories.<n>Then, the phase router allocates temporally consistent assignments to the same expert, allowing experts to preserve phase-specific expertise
  arXiv  Detail & Related papers  (2026-02-19T03:18:30Z)
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning [83.66308307152808]
  We propose StAbilized Mixture-of-Experts (SAME) for Multimodal Continual Instruction Tuning (MCIT)<n>SAME stabilizes expert selection by decomposing routing dynamics into subspaces and updating only task-relevant directions.<n>It also introduces adaptive expert activation to freeze selected experts during training, reducing redundant and cross-task interference.
  arXiv  Detail & Related papers  (2026-02-02T11:47:06Z)
• RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Merging [69.2230254959204]
  We propose RouteMark, a framework for IP protection in merged MoE models.<n>Our key insight is that task-specific experts exhibit stable and distinctive routing behaviors under probing inputs.<n>For attribution and tampering detection, we introduce a similarity-based matching algorithm.
  arXiv  Detail & Related papers  (2025-08-03T14:51:58Z)
• Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization [51.562474873972086]
  Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data.<n>Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt.<n>We propose TRIP, a Token-level prompt mixture with parameter-free routing framework for FedDG.
  arXiv  Detail & Related papers  (2025-04-29T11:06:03Z)
• Predicting Multi-Agent Specialization via Task Parallelizability [8.465921582175426]
  We present a closed-form bound that predicts when specialization improves performance depending on task regimes and team size.<n>We validate our model on two standard MARL benchmarks that represent opposite benchmarks.<n>Three follow-up experiments in Overcooked-AI demonstrate that the model works in environments with more complex spatial and resource bottlenecks.
  arXiv  Detail & Related papers  (2025-03-19T21:33:48Z)
• MoEC: Mixture of Expert Clusters [93.63738535295866]
  Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation.
  arXiv  Detail & Related papers  (2022-07-19T06:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.