Expert Routing for Communication-Efficient MoE via Finite Expert Banks

• URL: http://arxiv.org/abs/2605.05278v1
• Date: Wed, 06 May 2026 16:00:46 GMT
• Title: Expert Routing for Communication-Efficient MoE via Finite Expert Banks
• Abstract summary: Resource-efficient machine learning increasingly uses sparse Mixture-of-Experts (MoE) architectures.<n>We treat the gate as a channel and use $I(X;T)$ to quantify the routing information available to the selected expert.<n>The proposed framework provides a practical tool for analyzing resource-aware MoE inference systems.
• Score: 3.8673630752805437
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Resource-efficient machine learning increasingly uses sparse Mixture-of-Experts (MoE) architectures, where the gate acts as both a learning component and a routing interface controlling computation, communication, and accuracy. Motivated by finite-rate interpretations of MoE gating, we treat the gate as a stochastic channel and use $I(X;T)$ to quantify the routing information available to the selected expert. To make the associated information quantities tractable beyond synthetic examples, we develop a finite-bank MNIST construction using pretrained CNN experts and a discrete, data-dependent selection rule. Since the selected model belongs to a finite candidate set, the algorithmic mutual information $I(S;W)$ admits a closed-form discrete-entropy estimator from the empirical posterior $q(W|S)$. Sweeping a data-dependence parameter $α$, we observe that $\widehat I(S;W)$ monotonically tracks the generalization gap, while the Xu-Raginsky bound exhibits the expected looseness. We also compare with a uniform union-bound baseline and introduce an empirical estimator of $I(X;T)$ together with a Blahut-Arimoto procedure for tracing an accuracy-rate curve over the expert bank. The proposed framework provides a practical tool for analyzing resource-aware MoE inference systems and for interpreting $I(X;T)$ and $D(R_g)$ as design proxies for efficient expert routing.

Related papers

• Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation [64.51088225051959]
  Heterogeneous AI systems can improve quality and efficiency by routing queries to the specialist who can answer most effectively at the lowest cost.<n>We formalize this tradeoff as an instance of Pandora's Box, the classical problem of optimal search with costly inspection.<n>Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that determine, for each specialist and input, whether refining the value estimate is worth its cost.
  arXiv  Detail & Related papers  (2026-08-20T17:54:37Z)
• ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts [32.864092506780004]
  Mixture-of-Experts (MoE) models scale by activating only a small subset of experts per token.<n>We introduce ProbMoE, a probabilistic routing framework that models expert selection as a distribution over cardinality-constrained expert subsets.
  arXiv  Detail & Related papers  (2026-06-01T00:16:36Z)
• Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs [3.8673630752805437]
  Mixture-of-Experts (MoE) architectures decompose prediction tasks into specialized expert sub-networks selected by a gating mechanism.<n>We specialize a mutual-information bound and develop a rate-distortion characterization $D(R_g)$ of finite-rate gating.<n>The analysis yields capacity-aware limits for communication-constrained MoE systems, and numerical simulations on synthetic multi-expert models empirically confirm the predicted trade-offs between gating rate, expressivity, and generalization.
  arXiv  Detail & Related papers  (2026-02-16T17:26:12Z)
• Improving MoE Compute Efficiency by Composing Weight and Data Sparsity [50.654297246411545]
  Mixture-of-Experts layers achieve compute efficiency through weight sparsity.<n>Data sparsity, where each expert processes only a subset of tokens, offers a complementary axis.
  arXiv  Detail & Related papers  (2026-01-21T18:53:58Z)
• Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts [32.65737144630759]
  Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric "router" to dispatch tokens to a sparse subset of experts.<n>We introduce kNN-MoE, a retrieval-augmented routing framework that reuses optimal expert assignments from a memory of similar past cases.<n>Experiments show kNN-MoE outperforms zero-shot baselines and rivals computationally expensive supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-01-05T14:16:11Z)
• On the sample complexity of semi-supervised multi-objective learning [17.947890912560162]
  In multi-objective learning, several possibly competing prediction tasks must be solved jointly by a single model.<n>For objectives defined with Bregman losses, we prove that the complexity of $mathcalG$ may come into play only in terms of unlabeled data.<n>These rates are achieved by a simple, semi-supervised algorithm via pseudo-labeling.
  arXiv  Detail & Related papers  (2025-08-23T22:15:36Z)
• Probabilistically Tightened Linear Relaxation-based Perturbation Analysis for Neural Network Verification [83.25968588249776]
  We present a novel framework that combines over-approximation techniques from LiRPA-based approaches with a sampling-based method to compute tight intermediate reachable sets.<n>With negligible computational overhead, $textttPT-LiRPA$ exploiting the estimated reachable sets, significantly tightens the lower and upper linear bounds of a neural network's output.
  arXiv  Detail & Related papers  (2025-07-07T18:45:53Z)
• Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization [72.69498649272347]
  conditional distributions is a central problem in machine learning.<n>We propose a new paradigm that integrates both paired and unpaired data.<n>We show that our approach can theoretically recover true conditional distributions with arbitrarily small error.
  arXiv  Detail & Related papers  (2024-10-03T16:12:59Z)
• Estimating Treatment Effects under Recommender Interference: A Structured Neural Networks Approach [32.55211271796683]
  We show that commonly adopted difference-in-means estimators can lead to severely biased estimates due to recommender interference.<n>We propose a recommender choice model'' that explicitly represents the interference pathway.<n>We validate our method with a large-scale field experiment on Weixin short-video platform.
  arXiv  Detail & Related papers  (2024-06-20T14:53:26Z)
• Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks [74.68583356645276]
  In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis. We show for the first time that pMoE provably reduces the required number of training samples to achieve desirable generalization.
  arXiv  Detail & Related papers  (2023-06-07T00:16:10Z)
• FeDXL: Provable Federated Learning for Deep X-Risk Optimization [105.17383135458897]
  We tackle a novel federated learning (FL) problem for optimizing a family of X-risks, to which no existing algorithms are applicable. The challenges for designing an FL algorithm for X-risks lie in the non-decomability of the objective over multiple machines and the interdependency between different machines.
  arXiv  Detail & Related papers  (2022-10-26T00:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.