GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems

• URL: http://arxiv.org/abs/2605.19945v1
• Date: Tue, 19 May 2026 15:01:39 GMT
• Title: GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
• Abstract summary: Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per token.<n>We propose GEM, a framework for GPU variability-aware expert to GPU mapping for MoE models.<n>Our experiments show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline.
• Score: 2.7564583149786075
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per token. MoE serving engines distribute experts across multiple GPUs and route tokens to appropriate GPUs at inference time based on experts activated. They process tokens in lock-step fashion, where tokens within a batch must finish processing before proceeding to the next layer. This synchronization barrier acts as a critical bottleneck because the performance of MoE models is limited by the straggler GPU that finishes last. Stragglers emerge when too many heavily used experts are placed on the same GPU or the slowest GPU. While prior works place experts that balance token loads across GPUs, they all overlook GPU variability and often place highly used experts on the slowest GPUs. We propose GEM, GPU-variability-aware Expert Mapping, a framework for GPU variability-aware expert to GPU mapping for MoE models. GEM exploits two insights. First, we must place experts such that each GPU receives non-uniform token loads based on their variability and they all finish processing a layer at about the same time. Our studies show that there are two types of experts: consistent that are used most of the time and temporal that are often used together for the remaining time. Our second insight is that we must place simultaneously used consistent and temporal experts on different GPUs and avoid placing them on slower GPUs to reduce slowdown. GEM gathers the variability profile of GPUs for each model and task and uses the token load distributions per task to map experts to GPUs. Our experiments show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline.

Related papers

• GPU-Accelerated Algorithms for Graph Vector Search: Taxonomy, Empirical Study, and Research Directions [54.570944939061555]
  We present a comprehensive study of GPU-accelerated graph-based vector search algorithms.<n>We establish a detailed taxonomy of GPU optimization strategies and clarify the mapping between algorithmic tasks and hardware execution units.<n>Our findings offer clear guidelines for designing scalable and robust GPU-powered approximate nearest neighbor search systems.
  arXiv  Detail & Related papers  (2026-02-10T16:18:04Z)
• Scaling Behaviors of Evolutionary Algorithms on GPUs: When Does Parallelism Pay Off? [43.96509049196842]
  Evolutionary algorithms (EAs) are increasingly implemented on graphics processing units (GPUs) to leverage parallel processing capabilities for enhanced efficiency.<n>We investigate how GPU parallelism alters the behavior of EAs beyond simple acceleration metrics.<n>Our results reveal that the impact of GPU acceleration is highly heterogeneous and depends strongly on algorithmic structure.
  arXiv  Detail & Related papers  (2026-01-26T12:55:21Z)
• HeterMoE: Efficient Training of Mixture-of-Experts Models on Heterogeneous GPUs [26.00138507762884]
  HeterMoE is a system to efficiently train MoE models on heterogeneous GPUs.<n>We show that HeterMoE achieves up to 2.3x speed-up compared to existing MoE training systems.
  arXiv  Detail & Related papers  (2025-04-04T18:55:52Z)
• Mixture of Lookup Experts [63.787712153454464]
  Mixture-of-Experts (MoE) activates only a subset of experts during inference.<n>MoLE is a new MoE architecture that is efficient in both communication and VRAM usage.
  arXiv  Detail & Related papers  (2025-03-20T02:31:57Z)
• Forecasting GPU Performance for Deep Learning Training and Inference [10.741682409837612]
  NeuSight is a framework to predict the performance of various deep learning models, for both training and inference, on unseen GPUs without requiring actual execution.<n>NeuSight decomposes a single deep learning kernel prediction into smaller working sets called tiles, which are executed independently on the GPU.<n>It reduces the percentage error from 121.4% and 30.8% to 2.3% in predicting the latency of GPT3 model for training and inference on H100, compared to state-of-the-art prior work.
  arXiv  Detail & Related papers  (2024-07-18T18:47:52Z)
• Turn Waste into Worth: Rectifying Top-$k$ Router of MoE [111.12838294273033]
  MoE models are popular for training large language models due to their computational efficiency. The commonly used top-$k$ routing mechanism suffers from redundancy and memory costs due to the unbalanced routing. To address the dropped tokens and padding, we propose the Rectify-ify, comprising the Intra-GPU Rectification and the Fill-in Rectification. The combination of them achieves superior performance, surpassing the accuracy of the vanilla top-1 router by 4.7%.
  arXiv  Detail & Related papers  (2024-02-17T06:23:27Z)
• Benchmarking GPUs on SVBRDF Extractor Model [0.0]
  In this work, we try to differentiate the performance of different GPUs on neural network models that operate on bigger input images (256x256) In this work, we tried to differentiate the performance of different GPUs on neural network models that operate on bigger input images (256x256)
  arXiv  Detail & Related papers  (2023-10-19T17:09:06Z)
• An Analysis of Collocation on GPUs for Deep Learning Training [0.0]
  Multi-Instance GPU (MIG) is a new technology introduced by NVIDIA that can partition a GPU to better-fit workloads. In this paper, we examine the performance of a MIG-enabled A100 GPU under deep learning workloads containing various sizes and combinations of models.
  arXiv  Detail & Related papers  (2022-09-13T14:13:06Z)
• SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs [147.73127662757335]
  We present scalable Multi-hOp REasoning (SMORE), the first general framework for both single-hop and multi-hop reasoning in Knowledge Graphs (KGs) Using a single machine SMORE can perform multi-hop reasoning in Freebase KG (86M entities, 338M edges), which is 1,500x larger than previously considered KGs. SMORE increases throughput (i.e., training speed) over prior multi-hop KG frameworks by 2.2x with minimal GPU memory requirements.
  arXiv  Detail & Related papers  (2021-10-28T05:02:33Z)
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers [65.60007071024629]
  We show that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy. We show experimentally that Adaptive SGD outperforms four state-of-the-art solutions in time-to-accuracy.
  arXiv  Detail & Related papers  (2021-10-13T20:58:15Z)
• RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks with Fine-Grain Utilization [5.02836935036198]
  We propose RTGPU, which can schedule the execution of multiple GPU applications in real-time to meet hard deadlines. Our approach provides superior schedulability compared with previous work, and gives real-time guarantees to meet hard deadlines for multiple GPU applications.
  arXiv  Detail & Related papers  (2021-01-25T22:34:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.