Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference

• URL: http://arxiv.org/abs/2606.01007v1
• Date: Sun, 31 May 2026 04:51:09 GMT
• Title: Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference
• Abstract summary: We propose emphTask-Aware Coactivation Grouping (TACG) to group experts by task-aware co-activation.<n>We also introduce emphGeneric Expert Shared Replication (GESR) to identify generic experts with consistently central co-activation profiles.<n>Our framework reduces the average communication cost by 31.39% over the baseline, while preserving an average Jain fairness index of 0.9975.
• Score: 16.917643169487402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparsely activated Mixture-of-Experts (MoE) models scale capacity via conditional computation, but distributed inference suffers from cross-GPU expert communication and routing-induced load imbalance. Existing placement methods reduce this cost by co-locating frequently co-activated experts; however, they derive a single deployment plan from globally aggregated routing traces, thereby averaging away the heterogeneous, task-specific co-activation patterns that actually drive communication in multi-task serving. We observe that expert co-activation is strongly task-conditioned: pairs tightly coupled in one task family are often uncorrelated in another, so effective deployment should group experts by task-aware co-activation rather than by a task-agnostic average. Based on this insight, we propose \emph{Task-Aware Coactivation Grouping} (TACG), a deployment-time framework that uses family-specific dispatch and co-activation traces to derive per-expert task-family preferences, reweights the co-activation graph so that intra-family locality dominates grouping, and assigns each expert to a primary GPU under exact capacity constraints. To keep the static placement robust under online workload skew, we further introduce \emph{Generic Expert Shared Replication} (GESR), a lightweight companion that identifies generic experts with consistently central co-activation profiles, replicates them across a small set of secondary GPUs, and applies locality- and load-aware selection at serving time. Experiments on three representative open-source MoE models demonstrate that our framework reduces the average communication cost by 31.39\% over the baseline, while preserving an average Jain fairness index of 0.9975. This advantage persists even under severe distribution shifts in the inference data, consistently outperforming strong baselines.

Related papers

• FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA [15.068543696107227]
  Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples.<n>Existing methods typically specialize at client granularity, implicitly assuming task-coherent clients.<n>We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization.
  arXiv  Detail & Related papers  (2026-07-29T08:47:58Z)
• Coordination Graphs for Constrained Multi-Agent Reinforcement Learning [4.278821961637313]
  Coordination Graphs for Constrained Multi-Agent Reinforcement Learning (CG-CMARL)<n>System decomposes the joint problem into pairwise regions, each served by a set of shared Q-functions.<n>At execution time, Max-Sum message passing coordinates actions across the factor graph, while a Lagrangian multiplier controls the objective--constraint tradeoff.
  arXiv  Detail & Related papers  (2026-06-01T14:46:16Z)
• Improving the Efficiency of Language Agent Teams with Adaptive Task Graphs [52.26652574704317]
  Large language models (LLMs) are increasingly deployed in teams, yet existing coordination approaches often occupy two extremes.<n>We introduce Language Agent Teams for Task Evolution (LATTE), a framework for coordinating LLM teams inspired by distributed systems.
  arXiv  Detail & Related papers  (2026-05-07T14:19:17Z)
• Multi-Domain Learning with Global Expert Mapping [102.62297074508147]
  Mixture-of-Experts (MoE) models provide a scalable solution by routing inputs to specializedworks (experts)<n>We propose GEM (Global Expert Mapping), a planner-compiler framework that replaces the learned router with a global scheduler.<n>Our planner, based on linear programming relaxation, computes a fractional assignment of datasets to experts, while the compiler applies hierarchical rounding to convert this soft plan into a deterministic, capacity-aware mapping.<n>Experiments show that GEM-DINO achieves state-of-the-art performance on the UODB benchmark, with notable gains on underrepresented datasets and solves task interference in few
  arXiv  Detail & Related papers  (2026-04-20T21:09:34Z)
• GoAgent: Group-of-Agents Communication Topology Generation for LLM-based Multi-Agent Systems [55.12339141172908]
  GoAgent is a communication topology generation method that explicitly treats collaborative groups as the atomic units of MAS construction.<n>Experiments on six benchmarks demonstrate the state-of-the-art performance of GoAgent with 93.84% average accuracy while reducing token consumption by about 17%.
  arXiv  Detail & Related papers  (2026-03-20T06:21:32Z)
• C$^2$-Explorer: Contiguity-Driven Task Allocation with Connectivity-Aware Task Representation for Decentralized Multi-UAV Exploration [9.118945533198369]
  Efficient multi-temporal exploration under limited communication is severely bottlenecked by inadequate task representation and allocation.<n>We propose C$2$-Explorer, a decentralized framework that constructs a connectivity graph to decompose disconnected unknown components into independent task units.<n>We show that C$2$-Explorer consistently outperforms state-of-the-art (SOTA) baselines, reducing average exploration time by 43.1% and path length by 33.3%.
  arXiv  Detail & Related papers  (2026-03-08T15:54:33Z)
• Robust multi-task boosting using clustering and local ensembling [6.920276126310231]
  We propose Robust Multi-Task Boosting using Clustering and Local Ensembling (RMB-CLE)<n>RMB-CLE derives inter-task similarity directly from cross-task errors.<n>Within each cluster, a local ensemble enables robust knowledge sharing.
  arXiv  Detail & Related papers  (2026-02-15T16:59:23Z)
• MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching [60.886768806064936]
  Tool-Integrated Reasoning empowers large language models to tackle complex tasks by interleaving reasoning steps with external tool interactions.<n>Existing reinforcement learning methods rely on outcome- or trajectory-level rewards, assigning uniform advantages to all steps within a trajectory.<n>We propose MatchTIR, a framework that introduces fine-grained supervision via bipartite matching-based turn-level reward assignment and dual-level advantage estimation.
  arXiv  Detail & Related papers  (2026-01-15T18:59:23Z)
• Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling [6.549288471493216]
  We study a generic federated approximation problem involving $M$ agents.<n>The goal is for the agents to communicate intermittently via a server to find the root of the average of the agents' local operators.<n>We develop a novel algorithm titled texttFedHSA, and prove that it guarantees convergence to the correct point.
  arXiv  Detail & Related papers  (2025-04-15T22:13:55Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.