Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

• URL: http://arxiv.org/abs/2606.23739v1
• Date: Sun, 21 Jun 2026 13:43:27 GMT
• Title: Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search
• Abstract summary: We present an automated search pipeline for heterogeneous 4-Expert Mixture-of-Experts (MoE4)<n>The pipeline generated 4,463 candidate models across 197 batches, of which 1,021 were evaluated successfully.<n>A critical finding emerged from the campaign: due to alphabetical enumeration via itertools.
• Score: 44.70501115891534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present an automated large-scale search pipeline for heterogeneous 4-Expert Mixture-of-Experts (MoE4) architectures within the LEMUR neural network dataset ecosystem. Building on a hand-crafted heterogeneous MoE reference model, we replace manual design with a deterministic code-assembly generator that systematically combines base architecture families drawn from the LEMUR database into MoE4 ensembles, each governed by a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. Over a 28-day campaign on an NVIDIA RTX 4090, the pipeline generated 4,463 candidate models across 197 batches, of which 1,021 were evaluated successfully. A critical finding emerged from the campaign: due to alphabetical enumeration via itertools.combinations, the entire explored search space (4.8% of the theoretical 23,751 possible 4-family combinations) is anchored to a single family, AirNet. We characterise this coverage bias precisely, identify the root cause in the generator, and propose a stratified random sampling fix. Within the AirNet anchored scope, ShuffleNet and MobileNetV3 consistently co-produce the highest-accuracy ensembles (mean accuracy up to 0.632), while FractalNet and MNASNet are identified as low-yield families warranting exclusion in future campaigns. The pipeline, analysis artefacts, and corrected generator are released as part of the open-source NNGPT project at https://github.com/ABrain-One/nn-gpt

Related papers

• FANS: Federated Adaptive Network Search Learning for Heterogeneous Devices [44.62284429272935]
  Heterogeneous Learning (HFL) aims to train models across devices with diverse resource budgets while preserving data privacy.<n>We introduce Federated Adaptive Network Search (FANS), a hypernetwork-based framework that learns a shared architecture space.<n>We evaluate FANS on CIFAR-10, CIFAR-100, and MNLI using ResNet-18, DenseNet-121, and BERT-base, respectively.
  arXiv  Detail & Related papers  (2026-09-05T13:57:56Z)
• Scaling Closed-Loop Feature Channel Configuration with LLMs [48.83701310501069]
  The analysis covers 2000 generated candidates from 8 complete cycles, yielding 462 verified CIFAR-100 evaluations after task and metadata filtering.<n>The best model reaches 0.3676 with 11.8M parameters, compared with an early high-performing model at 0.3144 with 166.5M parameters.<n>Non-power-of-two channel widths occur in 41.8% of verified candidates, and the strongest models share structured channelallocation patterns.
  arXiv  Detail & Related papers  (2026-07-07T22:31:53Z)
• Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study [48.83701310501069]
  This paper investigates whether a single recipe is sufficient for heterogeneous architecture pools, or whether the optimal pairing varies across structurally diverse models.<n>We conduct a systematic empirical study of all $3 times 6 = 18$ combinations of six classifiers paired with three loss functions: Cross-Entropy (CEL), Negative Log-Likelihood (NLL), and the recently introduced genetically evolved NGL loss across the base models presented in LEMUR heterogeneous architecture pool on six image classification datasets.<n>Our results confirm that no single pairing is universally optimal. Cross-Entropy with Adam or AdamW is the most robust choice across
  arXiv  Detail & Related papers  (2026-06-18T20:51:42Z)
• StereoFactory: A Unified Merging Framework for Robust Stereo Matching [61.973843344605655]
  Stereo matching has advanced through foundation models trained on large-scale datasets, yet this paradigm suffers from a scalability bottleneck.<n>Model merging offers a scalable post-hoc alternative by integrating knowledge from specialized models after source checkpoints are available.<n>We propose StereoFactory, a coarse-to-fine evolutionary framework for adaptive model merging.
  arXiv  Detail & Related papers  (2026-06-16T03:36:59Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• DEI: Diversity in Evolutionary Inference for Quality-Diversity Search [0.14323566945483496]
  We present DEI: Diversity in Evolutionary Inference, a distributed Quality-Diversity (QD) search framework.<n>Unlike homogeneous parallel search, DEI treats each LLM's distinct creative prior as a complementary source of behavioral novelty.<n> nodes share local optimal solutions at the end of each round to seed the next round's population.
  arXiv  Detail & Related papers  (2026-05-26T15:00:57Z)
• A Hybrid Vision-Language Architecture for Automated Defect Reasoning and Report Generation in Industrial Inspection [0.42970700836450487]
  This paper describes a decoupled, edge-deployable pipeline for wind turbine blade inspection.<n>The Eyes a Y26-x-obb oriented bounding-box localizes defects at dataset-native resolution.<n>The Bridge a deterministic, parameter-free encoding module maps each detected bounding box to grid-referenced tokens.
  arXiv  Detail & Related papers  (2026-05-26T04:27:38Z)
• Global and Local Topology-Aware Attention with Persistent Homology and Euler Biases for Time-Series Forecasting [0.0]
  Time series often encode predictive geometric structure, including connectivity, cycles, shell-like geometry, directional changes, and nonlinear neighborhoods.<n>We introduce a topology-aware attention framework that adds such structure to attention logits using persistent homology (H0-H2)<n>We evaluate guarded topology-aware variants across three architecture families: lightweight attention/Ridge, PatchTSTForRegression, and TimeSeriesTransformerForPrediction.
  arXiv  Detail & Related papers  (2026-05-04T21:19:12Z)
• ARCS: Autoregressive Circuit Synthesis with Topology-Aware Graph Attention and Spec Conditioning [0.0]
  ARCS produces complete, SPICE-simulatable designs in milliseconds rather than the minutes required by search-based methods.<n>For single-model inference, a topology-aware Graph Transformer with Best-of-3 candidate selection reaches 85% simulation validity in 97ms, over 600x faster than random search.
  arXiv  Detail & Related papers  (2026-03-30T23:14:08Z)
• Evaluating Embedding Models and Pipeline Optimization for AI Search Quality [0.0]
  We evaluate the performance of various text embedding models and pipeline configurations for AI-driven search systems.<n>A custom evaluation dataset of 11,975 query-chunk pairs was synthesized from US City Council meeting transcripts.
  arXiv  Detail & Related papers  (2025-11-27T09:09:39Z)
• ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models [56.21470608621633]
  We propose a time estimation framework to decouple the architectural search from the target hardware. The proposed methodology extracts a set of models from micro- kernel and multi-layer benchmarks and generates a stacked model for mapping and network execution time estimation. We compare estimation accuracy and fidelity of the generated mixed models, statistical models with the roofline model, and a refined roofline model for evaluation.
  arXiv  Detail & Related papers  (2021-05-07T11:39:05Z)
• Inception Convolution with Efficient Dilation Search [121.41030859447487]
  Dilation convolution is a critical mutant of standard convolution neural network to control effective receptive fields and handle large scale variance of objects. We propose a new mutant of dilated convolution, namely inception (dilated) convolution where the convolutions have independent dilation among different axes, channels and layers. We explore a practical method for fitting the complex inception convolution to the data, a simple while effective dilation search algorithm(EDO) based on statistical optimization is developed.
  arXiv  Detail & Related papers  (2020-12-25T14:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.