Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search
Abstract Overview
This paper presents an automated pipeline for systematically generating and evaluating heterogeneous 4-expert mixture-of-experts (MoE4) vision models within the LEMUR ecosystem. The method replaces manual MoE design with deterministic code assembly, multi-stage validation, and fault-tolerant campaign automation for large-scale search over combinations of 29 base architecture families. In a 28-day campaign on CIFAR-10, the system generated 4,463 candidate models across 197 batches, with 1,021 models completing evaluation successfully. The study also analyzes failure modes and search-space behavior, showing that most failures were due to GPU memory limits and that the explored combinations were unintentionally biased by the enumeration scheme.
Novelty
The main novelty is an automated end-to-end pipeline for assembling heterogeneous 4-expert MoE architectures directly from a model database, including deterministic code generation, validation, and long-run campaign orchestration. A distinctive contribution is the explicit diagnosis of a search-space coverage bias caused by sorted combinatorial enumeration, along with a concrete randomization-based fix.
Results
The campaign successfully evaluated 1,021 MoE4 models, while revealing that 99.7% of failures were CUDA out-of-memory errors. Within the explored AirNet-anchored subset, the best model (AirNet + AlexNet + DPN68 + ResNet) reached 0.6801 top-1 accuracy on CIFAR-10 after one training epoch, and family-level analysis found ShuffleNet and MobileNetV3 to be the most favorable co-experts by mean accuracy.
Key Points
- The pipeline combines deterministic model assembly, syntax and forward-pass validation, deduplication, and resumable batch automation to support large-scale MoE4 exploration.
- Post hoc analysis showed that the search covered only 4.8% of the theoretical 23,751 combinations because lexicographic enumeration anchored every attempted configuration to the AirNet family.
- Among successful AirNet-anchored models, ShuffleNet and MobileNetV3 were associated with the strongest average accuracy, while FractalNet and MNASNet were poor candidates due to high failure rates or low performance utility.