DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models

• URL: http://arxiv.org/abs/2409.06669v1
• Date: Tue, 10 Sep 2024 17:36:15 GMT
• Title: DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
• Authors: Maryam Akhavan Aghdam, Hongpeng Jin, Yanzhao Wu,
• Abstract summary: We propose a novel Dynamically Allocates a variable number of experts for Mixture-of-Experts (DA-MoE) models based on an effective token importance measure. Our approach consistently outperforms the state-of-the-art Transformer based MoE model on the popular GLUE benchmark.
• Score: 1.4255659581428335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based Mixture-of-Experts (MoE) models have been driving several recent technological advancements in Natural Language Processing (NLP). These MoE models adopt a router mechanism to determine which experts to activate for routing input tokens. However, existing router mechanisms allocate a fixed number of experts to each token, which neglects the varying importance of different input tokens. In this study, we propose a novel dynamic router mechanism that Dynamically Allocates a variable number of experts for Mixture-of-Experts (DA-MoE) models based on an effective token importance measure. First, we show that the Transformer attention mechanism provides a natural and effective way of calculating token importance. Second, we propose a dynamic router mechanism that effectively decides the optimal number of experts (K) and allocates the top-K experts for each input token. Third, comprehensive experiments on several benchmark datasets demonstrate that our DA-MoE approach consistently outperforms the state-of-the-art Transformer based MoE model on the popular GLUE benchmark.

Related papers

• Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging [111.8456671452411]
  Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. We propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. We show that WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.
  arXiv  Detail & Related papers  (2024-10-29T07:16:31Z)
• Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast [58.98411447739218]
  Mixture-of-Experts (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. We propose Self-Contrast Mixture-of-Experts (SCMoE), a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. Our method is conceptually simple and computationally lightweight, as it incurs minimal latency compared to greedy decoding.
  arXiv  Detail & Related papers  (2024-05-23T12:45:29Z)
• Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models [33.834215393960605]
  We introduce the Dynamic Mixture of Experts (DynMoE) technique to enhance the efficiency of training and inference for Transformer-based foundational models. DynMoE incorporates a novel gating method that enables each token to automatically determine the number of experts to activate. Our results demonstrate the effectiveness of our approach to achieve competitive performance compared to GMoE for vision and language tasks, and MoE-LLaVA for vision-language tasks.
  arXiv  Detail & Related papers  (2024-05-23T08:18:30Z)
• Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training [73.90260246781435]
  We present Lory, the first approach that scales such architectures to autoregressive language model pre-training. We show significant performance gains over parameter-matched dense models on both perplexity and a variety of downstream tasks. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing.
  arXiv  Detail & Related papers  (2024-05-06T03:06:33Z)
• MMA-DFER: MultiModal Adaptation of unimodal models for Dynamic Facial Expression Recognition in-the-wild [81.32127423981426]
  Multimodal emotion recognition based on audio and video data is important for real-world applications. Recent methods have focused on exploiting advances of self-supervised learning (SSL) for pre-training of strong multimodal encoders. We propose a different perspective on the problem and investigate the advancement of multimodal DFER performance by adapting SSL-pre-trained disjoint unimodal encoders.
  arXiv  Detail & Related papers  (2024-04-13T13:39:26Z)
• Harder Tasks Need More Experts: Dynamic Routing in MoE Models [58.18526590138739]
  We introduce a novel dynamic expert selection framework for Mixture of Experts (MoE) models. Our method dynamically selects experts based on the confidence level in expert selection for each input.
  arXiv  Detail & Related papers  (2024-03-12T13:41:15Z)
• Routers in Vision Mixture of Experts: An Empirical Study [26.51711534240885]
  Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. Key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embeddings (tokens)
  arXiv  Detail & Related papers  (2024-01-29T08:58:07Z)
• Mixture of Attention Heads: Selecting Attention Heads Per Token [40.04159325505842]
  Mixture of Attention Heads (MoA) is a new architecture that combines multi-head attention with the MoE mechanism. MoA achieves stronger performance than the standard multi-head attention layer. MoA also automatically differentiates heads' utilities, providing a new perspective to discuss the model's interpretability.
  arXiv  Detail & Related papers  (2022-10-11T04:54:05Z)
• Taming Sparsely Activated Transformer with Stochastic Experts [76.0711573018493]
  Sparsely activated models (SAMs) can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. In this paper, we propose a new expert-based model, THOR (Transformer witH StOchastic ExpeRts) Unlike classic expert-based models, such as the Switch Transformer, experts in THOR are randomly activated for each input during training and inference.
  arXiv  Detail & Related papers  (2021-10-08T17:15:47Z)
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts [29.582683923988203]
  Mixture of Experts (MoE) based Transformer has shown promising results in many domains. In this work, we explore the MoE based model for speech recognition, named SpeechMoE. New router architecture is used in SpeechMoE which can simultaneously utilize the information from a shared embedding network.
  arXiv  Detail & Related papers  (2021-05-07T02:38:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.