Mixtral of Experts

• URL: http://arxiv.org/abs/2401.04088v1
• Date: Mon, 8 Jan 2024 18:47:34 GMT
• Title: Mixtral of Experts
• Abstract summary: Mixtral 8x7B is a Sparse Mixture of Experts (SMoE) language model. Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks.
• Score: 57.411379935325435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

Related papers

• Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs [8.622261628194893]
  Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token sparse.<n>We propose MetaNet, a support-set controller that predicts, for each layer, an expert-retention threshold and a bounded routing bias.
  arXiv  Detail & Related papers  (2026-08-27T06:01:46Z)
• Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts [0.0]
  We study whether these two roles, dispatch and aggregation, should be coupled.<n>A structured oracle improves full-horizon cross-entropy by 0.0160 +/- 0.0039 across three seeds.<n>FDAA improves fresh WikiText-103 test by Delta CE = -0.1523 +/- 0.0031 across three seeds.
  arXiv  Detail & Related papers  (2026-08-09T18:31:16Z)
• Mellum2 Technical Report [1.1239988514721222]
  Mellum 2 is a general-purpose language model specialized in software engineering.<n>It is the successor to the completion-focused 4B dense Mellum model.<n>Mellum 2 is competitive with open-weight baselines in the 4B-14B range while running at the per-token compute of a 2.5B dense model.
  arXiv  Detail & Related papers  (2026-05-29T13:01:11Z)
• dMoE: dLLMs with Learnable Block Experts [71.572316901001]
  We propose dMoE, a simple yet effective block-level MoE framework.<n>We show that dMoE substantially reduces the number of uniquely activated experts during inference.<n>It also reduces memory usage by 76.64% to 79.84% and achieves 1.14$times$ to 1.66$times$ end-to-end latency speedup.
  arXiv  Detail & Related papers  (2026-05-29T06:03:50Z)
• Pruning and Distilling Mixture-of-Experts into Dense Language Models [13.03679207895992]
  Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models.<n>We present the first systematic framework for converting a trained MoE into a standard fully dense architecture.<n>We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations.
  arXiv  Detail & Related papers  (2026-05-27T09:27:36Z)
• Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image [58.14192385042352]
  We introduce Multimodal RewardBench 2 (MMRB2), the first benchmark for reward models on multimodal understanding and (interleaved) generation.<n>MMRB2 spans four tasks: text-to-image, image editing, interleaved generation, and multimodal reasoning.<n>It provides 1,000 expert-annotated preference pairs per task from 23 models and agents across 21 source tasks.
  arXiv  Detail & Related papers  (2025-12-18T18:56:04Z)
• MixtureKit: A General Framework for Composing, Training, and Visualizing Mixture-of-Experts Models [6.3350351413826544]
  We introduce a modular open-source framework for constructing, training, and analyzing Mixture-of-Experts (MoE) models from arbitrary pre-trained or fine-tuned models.<n> MixtureKit supports three complementary methods: (i) emphTraditional MoE, which uses a single router per transformer block to select experts, (ii) emphBTX (Branch-Train-Mix), which introduces separate routers for each specified sub-layer enabling fine-grained token routing.
  arXiv  Detail & Related papers  (2025-12-13T01:22:10Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models [58.54288496296157]
  Chain-of-Experts (CoE) is a new Mixture-of-Experts (MoE) architecture that introduces sequential expert communication within each layer.<n>To support dynamic expert selection across iterations, CoE employs a dedicated router at each step within a layer.
  arXiv  Detail & Related papers  (2025-06-23T02:15:43Z)
• Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity [105.54207710201183]
  MoGE constrains tokens to activate an equal number of experts within each predefined expert group.<n>Pangu Pro MoE achieves 1148 tokens/s per card and can be further improved to 1528 tokens/s per card by speculative acceleration.
  arXiv  Detail & Related papers  (2025-05-27T16:40:21Z)
• FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models [19.984973014373118]
  We release FLAME-MoE, a completely open-source research suite composed of seven decoder-only models.<n>FLAME-MoE improves average accuracy by up to 3.4 points over dense baselines trained with identical FLOPs.
  arXiv  Detail & Related papers  (2025-05-26T17:06:25Z)
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts [63.67734699877724]
  MoE++ is a general and heterogeneous MoE framework that integrates both Feed-Forward Network(FFN) and zero-computation experts. MoE++ achieves better performance while delivering 1.1-2.1x expert forward throughput compared to a vanilla MoE model of the same size.
  arXiv  Detail & Related papers  (2024-10-09T18:01:27Z)
• MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts [95.26323548734692]
  MoMa is a modality-aware mixture-of-experts architecture for pre-training mixed-modal, early-fusion language models. Under a 1-trillion-token training budget, the MoMa 1.4B model, featuring 4 text experts and 4 image experts, achieves impressive FLOPs savings.
  arXiv  Detail & Related papers  (2024-07-31T17:46:51Z)
• Yuan 2.0-M32: Mixture of Experts with Attention Router [30.8849836244273]
  Yuan 2.0-M32 demonstrates competitive capability on coding, math, and various domains of expertise, with only 3.7B active parameters of 40B in total. Yuan 2.0-M32 surpass Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8 respectively.
  arXiv  Detail & Related papers  (2024-05-28T09:05:08Z)
• Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM [81.18305296110853]
  We investigate efficient methods for training Large Language Models (LLMs) to possess capabilities in multiple specialized domains. Our method, named Branch-Train-MiX (BTX), starts from a seed model, which is branched to train experts in embarrassingly parallel fashion. BTX generalizes two special cases, the Branch-Train-Merge method, which does not have the MoE finetuning stage to learn routing, and sparse upcycling, which omits the stage of training experts asynchronously.
  arXiv  Detail & Related papers  (2024-03-12T16:54:58Z)
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models [26.447210565680116]
  We propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts. We show that DeepSeekMoE achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation.
  arXiv  Detail & Related papers  (2024-01-11T17:31:42Z)
• Mistral 7B [62.17530433867458]
  Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks.
  arXiv  Detail & Related papers  (2023-10-10T17:54:58Z)
• Mixture-of-Experts with Expert Choice Routing [44.777850078713634]
  Prior work allocates a fixed number of experts to each token using a top-k function. We propose a heterogeneous mixture-of-experts employing an expert choice method. Our method improves training convergence time by more than 2x.
  arXiv  Detail & Related papers  (2022-02-18T17:46:11Z)
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference [17.97893143555333]
  Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. In this work, we investigate routing strategies at different granularity (token, sentence, task) in MoE models to bypass distillation. Experiments on WMT and a web-scale dataset suggest that task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy sub-networks from large sparse models.
  arXiv  Detail & Related papers  (2021-09-24T20:42:16Z)
• A Mixture of $h-1$ Heads is Better than $h$ Heads [63.12336930345417]
  We propose the mixture of attentive experts model (MAE) Experiments on machine translation and language modeling show that MAE outperforms strong baselines on both tasks. Our analysis shows that our model learns to specialize different experts to different inputs.
  arXiv  Detail & Related papers  (2020-05-13T19:05:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.