Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling

• URL: http://arxiv.org/abs/2603.27624v1
• Date: Sun, 29 Mar 2026 10:44:55 GMT
• Title: Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
• Abstract summary: Mixture-of-Experts is a promising approach for edge AI with low-batch inference.<n>MoE sparsity and dynamic gating shift distributed strategies toward much finer runtime granularity.<n>Fully Sharded Expert Data Parallelism is a parallelization paradigm specifically architected for low-batch MoE computation.
• Score: 29.998645673111536
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Mixture-of-Experts is a promising approach for edge AI with low-batch inference. Yet, on-device deployments often face limited on-chip memory and severe workload imbalance; the prevalent use of offloading further incurs off-chip memory access bottlenecks. Moreover, MoE sparsity and dynamic gating shift distributed strategies toward much finer granularity and introduce runtime scheduling considerations. Recently, high die-to-die bandwidth chiplet interconnects have created new opportunities for multi-chiplet systems to address workload imbalance and offloading bottlenecks with fine-grained scheduling. In this paper, we propose Fully Sharded Expert Data Parallelism, a parallelization paradigm specifically architected for low-batch MoE inference on multi-chiplet accelerators. FSE-DP attains adaptive computation-communication overlap and balanced load by orchestrating fine-grained, complementary expert streams along dynamic trajectories across high-bandwidth D2D links. The attendant dataflow complexity is tamed by a minimal, hardware-amenable set of virtualization rules and a lightweight scheduling algorithm. Our approach achieves 1.22 to 2.00 times speedup over state-of-the-art baselines and saves up to 78.8 percent on-chip memory.

Related papers

• Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs [22.6523068033902]
  Para-Pipe is a hierarchical mapping framework that integrates intra- and inter-stage operator parallelism within a pipelined architecture.<n>Para-Pipe navigates the trade-off between throughput and latency by selectively fine-tuning parallelism levels within and across pipeline stages.
  arXiv  Detail & Related papers  (2026-09-03T17:53:44Z)
• SynerDiff: Synergetic Continuous Batching for Fast and Parallel Diffusion Model Inference [8.251374398589906]
  Existing continuous methods suffer from severe resource contention during UNet-VAE.<n>We propose SynerDiff, an efficient continuous system built on intra-inter level synergy.
  arXiv  Detail & Related papers  (2026-05-09T09:39:39Z)
• Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference [36.81958522560282]
  We introduce Flux Attention, a context-aware framework that dynamically optimize attention at the layer level.<n>As a parameter-efficient approach, our framework requires only 12 hours of training on 8$times$A800 GPUs.<n>Our framework achieves a superior trade-off between performance and inference speed compared with baseline models, with speed improvements of up to $2.8times$ and $2.0times$ in the prefill and decode stages.
  arXiv  Detail & Related papers  (2026-04-08T07:36:17Z)
• AdaFuse: Accelerating Dynamic Adapter Inference via Token-Level Pre-Gating and Fused Kernel Optimization [84.25316984309725]
  Integration of dynamic, sparse structures with parameter-efficient adapters (e.g., LoRA) is a powerful technique for enhancing Large Language Models (LLMs)<n>Despite minimal increases in computational load, the computation latency often skyrockets, leading to decoding speeds slowing by over 2.5 times.<n>AdaFuse is a framework built on a tight co-design between the algorithm and the underlying hardware system to enable efficient dynamic adapter execution.
  arXiv  Detail & Related papers  (2026-03-12T12:46:42Z)
• TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Training [9.859893936091813]
  Training large language models (LLMs) is fundamentally constrained by limited device memory and costly inter-device communication.<n>We propose TawPipe, which exploits hierarchical bandwidth in distributed clusters for improved communication efficiency.
  arXiv  Detail & Related papers  (2025-11-12T21:06:37Z)
• Eliminating Multi-GPU Performance Taxes: A Systems Approach to Efficient Distributed LLMs [61.953548065938385]
  We introduce the ''Three Taxes'' (Bulk Synchronous, Inter- Kernel Data Locality, and Kernel Launch Overhead) as an analytical framework.<n>We propose moving beyond the rigid BSP model to address key inefficiencies in distributed GPU execution.<n>We observe a 10-20% speedup in end-to-end latency over BSP-based approaches.
  arXiv  Detail & Related papers  (2025-11-04T01:15:44Z)
• Taming the Tail: NoI Topology Synthesis for Mixed DL Workloads on Chiplet-Based Accelerators [8.493759841403682]
  Heterogeneous chiplet-based systems improve scaling by disag-gregating CPUs/GPUs and emerging technologies (HBM/DRAM)<n>However this on-package disaggregation introduces a latency inNetwork-on-Interposer(NoI)
  arXiv  Detail & Related papers  (2025-10-28T06:36:44Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs [81.5049387116454]
  We introduce APB, an efficient long-context inference framework.<n>APB uses multi-host approximate attention to enhance prefill speed.<n>APB achieves speeds of up to 9.2x, 4.2x, and 1.6x compared with FlashAttn, RingAttn, and StarAttn, respectively.
  arXiv  Detail & Related papers  (2025-02-17T17:59:56Z)
• InTAR: Inter-Task Auto-Reconfigurable Accelerator Design for High Data Volume Variation in DNNs [5.762543012823378]
  InTAR is a novel accelerator design methodology for HDV applications on FPGAs.<n>It switches execution patterns automatically with a static schedule determined before circuit design.<n>InTAR achieves a high clock frequency with fewer resources and low reconfiguration time.
  arXiv  Detail & Related papers  (2025-02-12T21:43:51Z)
• COMPASS: A Compiler Framework for Resource-Constrained Crossbar-Array Based In-Memory Deep Learning Accelerators [6.172271429579593]
  We propose a compiler framework for resource-constrained crossbar-based processing-in-memory (PIM) deep neural network (DNN) accelerators.<n>We propose an algorithm to determine the optimal partitioning that divides the layers so that each partition can be accelerated on chip.
  arXiv  Detail & Related papers  (2025-01-12T11:31:25Z)
• Real-Time GPU-Accelerated Machine Learning Based Multiuser Detection for 5G and Beyond [70.81551587109833]
  nonlinear beamforming filters can significantly outperform linear approaches in stationary scenarios with massive connectivity. One of the main challenges comes from the real-time implementation of these algorithms. This paper explores the acceleration of APSM-based algorithms through massive parallelization.
  arXiv  Detail & Related papers  (2022-01-13T15:20:45Z)
• EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inference [82.1584439276834]
  Transformer-based language models such as BERT provide significant accuracy improvement for a multitude of natural language processing (NLP) tasks. We present EdgeBERT, an in-depth algorithm- hardware co-design for latency-aware energy optimization for multi-task NLP.
  arXiv  Detail & Related papers  (2020-11-28T19:21:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.