The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism

• URL: http://arxiv.org/abs/2608.08650v1
• Date: Sun, 09 Aug 2026 11:46:29 GMT
• Title: The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
• Abstract summary: This survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts systems.<n>We describe eight architectural milestones as a dependency graph with six mainline developments and two branches, rather than as eight successive generations.<n>We conclude with equal-budget pretraining experiments, quality and systems metrics, and open research questions.
• Score: 3.3534582603188476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone. This technical survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure. We describe eight architectural milestones as a dependency graph with six mainline developments and two orthogonal branches, rather than as eight successive generations. We then analyze individual systems through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism. These planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation is mapped onto physical devices. The framework connects algorithmic choices such as Top-k routing, shared experts, fine-grained experts, and dynamic expert composition with systems concerns including token dispatch, device placement, all-to-all communication, and communication-computation overlap. We conclude with equal-budget pretraining experiments, quality and systems metrics, and open research questions. The main trend is a shift from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.

Related papers

• Unifying Graph Neural Networks Through a Common Layer Equation [75.88127999406392]
  We introduce a common layer equation that represents covered architectures through seven components.<n>The central factorization separates where information moves, encoded by the propagation bank, from what moves, encoded by the message maps.<n>We make this unification explicit and checkable through worked reductions of canonical layers and component assignments.
  arXiv  Detail & Related papers  (2026-08-17T04:37:59Z)
• Agentic Designer: Progressive Multi-Agent Collaboration for Structure-Aware Interior Layout Generation [84.5255199507027]
  Agentic Designer is a progressive framework that formulates structure-aware interior layout generation as an iterative and constraint-verified decision process.<n>By layout decomposing into modular stages of proposal, verification, synthesis and adjustment, the framework coordinates three specialized agents, a Generator, an Evaluator, and a Refiner.<n>This mechanism enforces stepwise geometric validation and correction before each placement is committed, thereby preventing error accumulation.
  arXiv  Detail & Related papers  (2026-07-23T02:46:11Z)
• From Prompt to Process: a Process Taxonomy and Comparative Assessment of Frameworks Supporting AI Software Development Agents [0.0]
  We focus on the operational frameworks that turn AI capabilities into process.<n>Each attack AI development through a different path: spec-driven development in full and lightweight variants, agent-driven agile planning, context engineering over the agent, worktree isolation and review, and recovery of operational specifications from legacy systems.<n>Our central contribution is a six-dimension process taxonomy: specification, context, roles, execution, validation and portability, with a scoring rubric that turns it into a replicable instrument.
  arXiv  Detail & Related papers  (2026-06-03T14:49:15Z)
• Architectural Design Decisions in AI Agent Harnesses [1.8222732878503212]
  This paper presents a protocol-guided, source-grounded empirical study of 70 publicly available agent-system projects.<n>We identify five recurring design dimensions (subagent architecture, context management, tool systems, safety mechanisms, and orchestration) and find that the corpus favors file-persistent, hybrid, and hierarchical context strategies.<n>We synthesize five recurring architectural patterns spanning lightweight tools, balanced CLI frameworks, multi-agent orchestrators, enterprise systems, and scenario-verticalized projects.
  arXiv  Detail & Related papers  (2026-04-20T10:39:34Z)
• A Two-Dimensional Framework for AI Agent Design Patterns: Cognitive Function and Execution Topology [54.72352855252792]
  We propose a two-dimensional classification that combines a Cognitive Function axis with seven categories and an Execution Topology axis with six structural archetypes.<n>The resulting 7x6 matrix identifies 27 named patterns, 13 with original names and provides a principled, framework-neutral, and model-agnostic vocabulary for AI agent architecture design.
  arXiv  Detail & Related papers  (2026-03-16T04:01:01Z)
• GenAI for Systems: Recurring Challenges and Design Principles from Software to Silicon [62.2138479061386]
  Generative AI is reshaping how computing systems are designed, optimized, and built, yet research remains fragmented across software, architecture, and chip design communities.<n>This paper takes a cross-stack perspective, examining how generative models are being applied from code generation and distributed runtimes through hardware design space exploration to RTL synthesis, physical layout, and verification.
  arXiv  Detail & Related papers  (2026-02-16T22:45:33Z)
• OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts Graph Generative Models [57.94189874119267]
  Multi-Agent Systems (MAS) offer a powerful paradigm for solving complex problems.<n>Current graph learning-based design methodologies often adhere to a "one-for-one" paradigm.<n>We propose OFA-TAD, a one-for-all framework that generates adaptive collaboration graphs for any task described in natural language.
  arXiv  Detail & Related papers  (2026-01-19T12:23:44Z)
• Revisiting the Ordering of Channel and Spatial Attention: A Comprehensive Study on Sequential and Parallel Designs [9.19832186254292]
  We build an evaluation suite of 18 topologies across four classes: sequential, parallel, multi-scale, and residual.<n>Across two vision and nine medical datasets, we uncover a "data scale-method-performance" coupling law.<n>We propose scenario-based guidelines for building future attention modules.
  arXiv  Detail & Related papers  (2026-01-12T08:32:37Z)
• Designing Domain-Specific Agents via Hierarchical Task Abstraction Mechanism [61.01709143437043]
  We introduce a novel agent design framework centered on a Hierarchical Task Abstraction Mechanism (HTAM)<n>Specifically, HTAM moves beyond emulating social roles, instead structuring multi-agent systems into a logical hierarchy that mirrors the intrinsic task-dependency graph of a given domain.<n>We instantiate this framework as EarthAgent, a multi-agent system tailored for complex geospatial analysis.
  arXiv  Detail & Related papers  (2025-11-21T12:25:47Z)
• CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning [81.08755597239262]
  Existing approaches suffer from a trade-off: generalist agents excel at planning but perform poorly in execution, while specialized agents demonstrate the opposite weakness.<n>Recent compositional frameworks attempt to bridge this gap by combining a planner and an actor, but they are typically static and non-trainable.<n>We introduce CODA, a novel and trainable compositional framework that integrates a generalist planner with a specialist executor.
  arXiv  Detail & Related papers  (2025-08-27T17:59:50Z)
• AI Accelerators for Large Language Model In-ference: Architecture Analysis and Scaling Strategies [10.520360508397237]
  Large-language models (LLMs) are driving a new wave of specialized hardware for inference.<n>This paper presents the first workload-centric, cross-architectural performance study of commercial AI accelerators.
  arXiv  Detail & Related papers  (2025-05-13T20:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.