Generalization and Scaling Laws for Mixture-of-Experts Transformers

• URL: http://arxiv.org/abs/2604.09175v1
• Date: Fri, 10 Apr 2026 09:59:48 GMT
• Title: Generalization and Scaling Laws for Mixture-of-Experts Transformers
• Abstract summary: We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers.<n>We derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead.<n>We prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers that cleanly separates \emph{active} per-input capacity from routing combinatorics. By conditioning on fixed routing patterns and union-bounding across them, we derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead. Combined with a standard ERM analysis for squared loss, this yields a generalization bound under a $d$-dimensional manifold data model and $C^β$ targets, showing that approximation and estimation trade off as in dense networks once active parameters are accounted for appropriately. We further prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts, depending on the dominant bottleneck. From these results we derive neural scaling laws for model size, data size, and compute-optimal tradeoffs. Overall, our results provide a transparent statistical reference point for reasoning about MoE scaling, clarifying which behaviors are certified by worst-case theory and which must arise from data-dependent routing structure or optimization dynamics.

Related papers

• Unlocking Feature Learning in Gated Delta Networks at Scale [52.705562336559446]
  Training and scaling Large Language Models demand enormous computational resources.<n>We derive the scaling rules for Gated Delta Network.<n> Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer.
  arXiv  Detail & Related papers  (2026-06-02T08:45:24Z)
• A renormalization-group inspired lattice-based framework for piecewise generalized linear models [0.6599344783327052]
  We introduce a class of models inspired by renormalization group (RG) theory.<n>They are built on additive hierarchical expansions analogous to those appearing in functional ANOVA and mixed-effects models.<n>We use techniques from statistical physics -- specifically replica analysis -- to study their generalization properties.
  arXiv  Detail & Related papers  (2026-05-06T22:27:50Z)
• Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization [19.86300998075755]
  We propose a reusable framework for holistic MoE architectural optimization.<n>We first show that FLOPs per token alone is an inadequate fairness metric for MoE models.<n>We then reduce the 16-dimensional architectural search space to two sequential low-dimensional phases.<n>Our framework yields robust scaling laws that map any compute budget to a complete, optimal MoE architecture.
  arXiv  Detail & Related papers  (2026-03-23T11:51:35Z)
• Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design [37.14769075463234]
  Mixture-of-Experts (MoE) models have emerged as an efficient method for scaling model capacity without proportionally increasing computation.<n>We define the ratio $r$ as the fraction of total FLOPs per token dedicated to the expert layers versus the attention layers.<n>Our analysis leads to an explicit formula for $r*$, enabling precise control over the expert-attention compute allocation.
  arXiv  Detail & Related papers  (2026-03-11T03:49:04Z)
• Understanding Model Merging: A Unified Generalization Framework for Heterogeneous Experts [36.26786113564521]
  Model merging efficiently aggregates capabilities from multiple fine-tuned models into a single one.<n>Despite empirical successes, a unified theory for its effectiveness under heterogeneous finetuning hyper parameters remains missing.<n>We use $L$-Stability theory to analyze the generalization of the merged model $boldsymbolx_avg$.
  arXiv  Detail & Related papers  (2026-01-29T13:22:06Z)
• Mixture-of-Experts Models in Vision: Routing, Optimization, and Generalization [0.0]
  We study MoE behavior in an image classification setting, focusing on predictive performance, expert utilization, and generalization.<n>We compare dense, SoftMoE, and SparseMoE classifier heads on the CIFAR10 dataset under comparable model capacity.<n>Both MoE variants achieve slightly higher validation accuracy than the dense baseline while maintaining balanced expert utilization through regularization.<n>We find that SoftMoE exhibits higher sharpness by these metrics, while Dense and SparseMoE lie in a similar curvature regime, despite all models achieving comparable generalization performance.
  arXiv  Detail & Related papers  (2026-01-21T14:22:25Z)
• Theoretical Foundations of Scaling Law in Familial Models [46.506708373314375]
  We introduce Granularity (G) as a fundamental scaling variable alongside model size (N) and training tokens (D)<n>Our results reveal that the granularity penalty follows a multiplicative power law with an extremely small exponent.<n> Practically, it validates the "train once, deploy many" paradigm, demonstrating that deployment flexibility is achievable.
  arXiv  Detail & Related papers  (2025-12-29T12:01:58Z)
• Towards a Comprehensive Scaling Law of Mixture-of-Experts [54.117786590884776]
  We propose a comprehensive and precise joint MoE scaling law that considers all essential factors.<n>Our results demonstrate that the optimal settings for $G$ and $S$ are independent of both the model architecture and data size.<n>Our proposed MoE scaling law could function as an accurate and insightful guidance to facilitate future MoE model design and training.
  arXiv  Detail & Related papers  (2025-09-28T06:35:34Z)
• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.<n>We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• Online Variational Sequential Monte Carlo [49.97673761305336]
  We build upon the variational sequential Monte Carlo (VSMC) method, which provides computationally efficient and accurate model parameter estimation and Bayesian latent-state inference. Online VSMC is capable of performing efficiently, entirely on-the-fly, both parameter estimation and particle proposal adaptation.
  arXiv  Detail & Related papers  (2023-12-19T21:45:38Z)
• ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
  We propose a new regularizer, namely, Equivariance Regularizer (ER) ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities. The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-24T08:18:05Z)
• Post-mortem on a deep learning contest: a Simpson's paradox and the complementary roles of scale metrics versus shape metrics [61.49826776409194]
  We analyze a corpus of models made publicly-available for a contest to predict the generalization accuracy of neural network (NN) models. We identify what amounts to a Simpson's paradox: where "scale" metrics perform well overall but perform poorly on sub partitions of the data. We present two novel shape metrics, one data-independent, and the other data-dependent, which can predict trends in the test accuracy of a series of NNs.
  arXiv  Detail & Related papers  (2021-06-01T19:19:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.