Exploring Sparse Expert Models and Beyond
- URL: http://arxiv.org/abs/2105.15082v2
- Date: Tue, 1 Jun 2021 15:28:15 GMT
- Title: Exploring Sparse Expert Models and Beyond
- Authors: An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang
Wang, Jie Zhang, Jiamang Wang, Yong Li, Di Zhang, Wei Lin, Lin Qu, Jingren
Zhou, Hongxia Yang
- Abstract summary: Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost.
We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing.
This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
- Score: 51.90860155810848
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Mixture-of-Experts (MoE) models can achieve promising results with outrageous
large amount of parameters but constant computation cost, and thus it has
become a trend in model scaling. Still it is a mystery how MoE layers bring
quality gains by leveraging the parameters with sparse activation. In this
work, we investigate several key factors in sparse expert models. We observe
that load imbalance may not be a significant problem affecting model quality,
contrary to the perspectives of recent studies, while the number of sparsely
activated experts $k$ and expert capacity $C$ in top-$k$ routing can
significantly make a difference in this context. Furthermore, we take a step
forward to propose a simple method called expert prototyping that splits
experts into different prototypes and applies $k$ top-$1$ routing. This
strategy improves the model quality but maintains constant computational costs,
and our further exploration on extremely large-scale models reflects that it is
more effective in training larger models. We push the model scale to over $1$
trillion parameters and implement it on solely $480$ NVIDIA V100-32GB GPUs, in
comparison with the recent SOTAs on $2048$ TPU cores. The proposed giant model
achieves substantial speedup in convergence over the same-size baseline.
Related papers
- Improving Large Models with Small models: Lower Costs and Better Performance [81.55672406002715]
We propose Data Shunt$+$ (DS$+$), a general paradigm for collaboration of small and large models.
For instance, ChatGPT achieves an accuracy of $94.43%$ on Amazon Product sentiment analysis, and DS$+$ achieves an accuracy of $95.64%$, while the cost has been reduced to only $31.18%$.
arXiv Detail & Related papers (2024-06-15T14:44:43Z) - Large Language Model Pruning [0.0]
We suggest a model pruning technique specifically focused on LLMs.
The proposed methodology emphasizes the explainability of deep learning models.
We also explore the difference between pruning on large-scale models vs. pruning on small-scale models.
arXiv Detail & Related papers (2024-05-24T18:22:15Z) - EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods.
EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
arXiv Detail & Related papers (2024-05-23T05:25:45Z) - A Large-Scale Exploration of $μ$-Transfer [0.0]
$mu$-Transfer yields scaling rules for model.
introductors and learning rates.
$mu$-Transfer is not yet widely adopted, perhaps due to higher implementation complexity, many variations, or complex theoretical background.
We study models of up to 10B parameters and training budgets of up to 190B tokens, and find $mu$-Transfer works as intended for the majority of important cases, yet also identify a few cases where it may not.
arXiv Detail & Related papers (2024-04-08T17:59:44Z) - PanGu-$\pi$: Enhancing Language Model Architectures via Nonlinearity
Compensation [97.78045712375047]
We present a new efficient model architecture for large language models (LLMs)
We show that PanGu-$pi$-7B can achieve a comparable performance to that of benchmarks with about 10% inference speed-up.
In addition, we have deployed PanGu-$pi$-7B in the high-value domains of finance and law, developing an LLM named YunShan for practical application.
arXiv Detail & Related papers (2023-12-27T11:49:24Z) - Anytime Model Selection in Linear Bandits [61.97047189786905]
We develop ALEXP, which has an exponentially improved dependence on $M$ for its regret.
Our approach utilizes a novel time-uniform analysis of the Lasso, establishing a new connection between online learning and high-dimensional statistics.
arXiv Detail & Related papers (2023-07-24T15:44:30Z) - Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion [54.33764537135906]
VideoQA Transformer models demonstrate competitive performance on standard benchmarks.
Do these models capture the rich multimodal structures and dynamics from video and text jointly?
Are they achieving high scores by exploiting biases and spurious features?
arXiv Detail & Related papers (2023-06-15T06:45:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.