MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale

• URL: http://arxiv.org/abs/2604.07030v1
• Date: Wed, 08 Apr 2026 12:47:57 GMT
• Title: MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
• Abstract summary: We propose the MoE Routing Testbed, a setup that gives clearer visibility into routing dynamics at small scale.<n>The testbed pairs a data mix with clearly distinguishable domains with a reference router that prescribes ideal routing.<n>We show that balancing scope is the crucial factor that allows specialization while maintaining high expert utilization.
• Score: 12.078286327859425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) architectures are increasingly popular for frontier large language models (LLM) but they introduce training challenges due to routing complexity. Fully leveraging parameters of an MoE model requires all experts to be well-trained and to specialize in non-redundant ways. Assessing this, however, is complicated due to lack of established metrics and, importantly, many routing techniques exhibit similar performance at smaller sizes, which is often not reflective of their behavior at large scale. To address this challenge, we propose the MoE Routing Testbed, a setup that gives clearer visibility into routing dynamics at small scale while using realistic data. The testbed pairs a data mix with clearly distinguishable domains with a reference router that prescribes ideal routing based on these domains, providing a well-defined upper bound for comparison. This enables quantifiable measurement of expert specialization. To demonstrate the value of the testbed, we compare various MoE routing approaches and show that balancing scope is the crucial factor that allows specialization while maintaining high expert utilization. We confirm that this observation generalizes to models 35x larger.

Related papers

• Multi-level context Modeling for consistent expert selection in Mixture-of-Experts [3.4913929232937604]
  Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts.<n>Existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers.<n>We propose Multi-level Context Fusion MOE, a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions.
  arXiv  Detail & Related papers  (2026-07-17T18:22:42Z)
• Multi-Domain Learning with Global Expert Mapping [102.62297074508147]
  Mixture-of-Experts (MoE) models provide a scalable solution by routing inputs to specializedworks (experts)<n>We propose GEM (Global Expert Mapping), a planner-compiler framework that replaces the learned router with a global scheduler.<n>Our planner, based on linear programming relaxation, computes a fractional assignment of datasets to experts, while the compiler applies hierarchical rounding to convert this soft plan into a deterministic, capacity-aware mapping.<n>Experiments show that GEM-DINO achieves state-of-the-art performance on the UODB benchmark, with notable gains on underrepresented datasets and solves task interference in few
  arXiv  Detail & Related papers  (2026-04-20T21:09:34Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMs [24.791817951102487]
  We show that aligning the manifold of routing weights with that of task embedding can effectively reduce the gap.<n>In experiments, we finetune routers in OLMoE, DeepSeekMoE, and Qwen3-MoE using RoMA.
  arXiv  Detail & Related papers  (2025-11-10T18:59:53Z)
• RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Merging [69.2230254959204]
  We propose RouteMark, a framework for IP protection in merged MoE models.<n>Our key insight is that task-specific experts exhibit stable and distinctive routing behaviors under probing inputs.<n>For attribution and tampering detection, we introduce a similarity-based matching algorithm.
  arXiv  Detail & Related papers  (2025-08-03T14:51:58Z)
• RouterRetriever: Routing over a Mixture of Expert Embedding Models [58.987116118425995]
  We introduce RouterRetriever, a retrieval model that leverages a mixture of domain-specific experts by using a routing mechanism.<n> RouterRetriever is the first work to demonstrate the advantages of routing over a mixture of domain-specific expert embedding models.
  arXiv  Detail & Related papers  (2024-09-04T13:16:55Z)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts [49.394145046409044]
  This paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models.
  arXiv  Detail & Related papers  (2024-05-26T17:52:58Z)
• Performance Characterization of Expert Router for Scalable LLM Inference [0.4726677580049183]
  Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains. deploying and serving these models at scale with optimal throughput and latency remains a significant challenge. This paper introduces Expert Router, a scalable routing architecture that directs to specialized expert models.
  arXiv  Detail & Related papers  (2024-04-22T16:33:42Z)
• Soft Merging of Experts with Adaptive Routing [38.962451264172856]
  We introduce Soft Merging of Experts with Adaptive Routing (SMEAR) SMEAR avoids discrete routing by using a single "merged" expert constructed via a weighted average of all of the experts' parameters. We empirically validate that models using SMEAR outperform models that route based on metadata or learn sparse routing through gradient estimation.
  arXiv  Detail & Related papers  (2023-06-06T15:04:31Z)
• MoEC: Mixture of Expert Clusters [93.63738535295866]
  Sparsely Mixture of Experts (MoE) has received great interest due to its promising scaling capability with affordable computational overhead. MoE converts dense layers into sparse experts, and utilizes a gated routing network to make experts conditionally activated. However, as the number of experts grows, MoE with outrageous parameters suffers from overfitting and sparse data allocation.
  arXiv  Detail & Related papers  (2022-07-19T06:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.