Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts

• URL: http://arxiv.org/abs/2608.08853v1
• Date: Sun, 09 Aug 2026 18:31:16 GMT
• Title: Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
• Abstract summary: We study whether these two roles, dispatch and aggregation, should be coupled.<n>A structured oracle improves full-horizon cross-entropy by 0.0160 +/- 0.0039 across three seeds.<n>FDAA improves fresh WikiText-103 test by Delta CE = -0.1523 +/- 0.0031 across three seeds.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) routers commonly use the same scores both to select experts and to weight their already-computed outputs. We study whether these two roles, dispatch and aggregation, should be coupled. On pretrained OLMoE-1B-7B, we keep selected Top-8 expert IDs, expert computation, and total selected router mass fixed and change only within-set aggregation. A structured oracle improves full-horizon cross-entropy by 0.0160 +/- 0.0039 across three seeds; the router's top-scored expert is the counterfactual-best vertex only 17.2% of the time, with router-utility Spearman 0.030. We therefore train Fixed-Dispatch Adaptive Aggregation (FDAA), a 301K-parameter post-compute head optimized directly with the language-modeling objective while freezing the backbone, router, and experts. On OLMoE, FDAA improves fresh WikiText-103 test by Delta CE = -0.1523 +/- 0.0031 across three seeds, and mixed-domain training gives robust gains on WikiText-103, C4, and held-out Penn Treebank under frozen confirmatory evaluation. We also replicate the fixed-dispatch audit on DeepSeek-V2-Lite, which uses Top-6 routed experts plus shared experts. Best-vertex headroom remains significant on WikiText and C4, while router Top1 identifies the best selected expert in only 12.5% and 16.7% of audited examples. In a one-seed mixed-domain replication, FDAA improves locked WikiText and PTB, while C4 is statistically neutral. These results support a cross-architecture distinction between expert selection and expert commitment.

Related papers

• ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs [15.24786135265898]
  Mixture-of-Experts (MoE) architectures provide an efficient paradigm for scaling large language models (LLMs)<n>We propose ACE, a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping.
  arXiv  Detail & Related papers  (2026-09-04T14:56:00Z)
• Training-Free Halving of Activated Experts in Fine-Grained Mixture-of-Experts Models [3.749501286035187]
  We show that renormalization implicitly calibrates expert output gain to the training top-$k$.<n>We find that removing renormalization entirely is catastrophic, showing that preserving a suitable reference mass is crucial.
  arXiv  Detail & Related papers  (2026-09-04T00:11:28Z)
• Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms [50.28963615440865]
  We organize three fusion paradigms by the artefacts they reuse.<n>We compare all three using shared experts and data across model scales and a multi-domain benchmark suite.
  arXiv  Detail & Related papers  (2026-08-27T17:38:04Z)
• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding [21.76650680783374]
  Speculative decoding verifies a tree of draft tokens in one target-model forward pass.<n>For a mixture-of-experts (MoE) target, parallel verification can activate the union of the experts selected by all tree nodes.<n>We introduce AcceptMoE, a verifier-side expert selector that automatically adjusts the number of eligible experts for each verification block.
  arXiv  Detail & Related papers  (2026-08-04T01:01:12Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection [0.0]
  CLEAR-MoE is a post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights.<n>On a GTX 960, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense routers.
  arXiv  Detail & Related papers  (2026-06-26T18:12:03Z)
• EEVEE: Towards Test-time Prompt Learning in the Real World for Self-Improving Agents [64.96332056338923]
  EEVEE is the first multi-dataset test-time prompt learning framework for LLM agents.<n>It enables test-time prompt learning under real-world task streams.<n>EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2.
  arXiv  Detail & Related papers  (2026-06-09T17:57:16Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• MoE-Sieve: Routing-Guided LoRA for Efficient MoE Fine-Tuning [0.5076419064097734]
  MoE-Sieve is a simple routing-guided framework for LoRA fine-tuning.<n>We show that tuning only the top 25% routed experts per layer remains competitive with full LoRA.<n>We also observe a non-monotonic relationship between expert count and seed-to-seed variance.
  arXiv  Detail & Related papers  (2026-03-25T07:53:09Z)
• Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition [22.046185271408202]
  Mixture-of-experts (MoE) architectures have expanded from language modeling to automatic speech recognition (ASR)<n>Our analysis reveals that routers in most layers make expert choices that are not strongly correlated with the choices of the routers in other layers.<n>To increase the cooperation between experts in different layers and encourage greater specialization, we use a shared router across different MoE layers.
  arXiv  Detail & Related papers  (2025-07-08T07:18:33Z)
• Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models [58.54288496296157]
  Chain-of-Experts (CoE) is a new Mixture-of-Experts (MoE) architecture that introduces sequential expert communication within each layer.<n>To support dynamic expert selection across iterations, CoE employs a dedicated router at each step within a layer.
  arXiv  Detail & Related papers  (2025-06-23T02:15:43Z)
• Mixtral of Experts [57.411379935325435]
  Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model. Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks.
  arXiv  Detail & Related papers  (2024-01-08T18:47:34Z)
• Mixture-of-Experts with Expert Choice Routing [44.777850078713634]
  Prior work allocates a fixed number of experts to each token using a top-k function. We propose a heterogeneous mixture-of-experts employing an expert choice method. Our method improves training convergence time by more than 2x.
  arXiv  Detail & Related papers  (2022-02-18T17:46:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.