HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference

• URL: http://arxiv.org/abs/2608.00577v1
• Date: Sat, 01 Aug 2026 10:30:01 GMT
• Title: HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference
• Abstract summary: HetRoute is a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference.<n>HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%.
• Score: 47.0879800334801
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models have become a dominant architecture for large-scale AI services, yet deploying them over geo-distributed heterogeneous edge servers remains challenging. When the Top-k activated experts of a token are spread across multiple servers, the optimal routing depends jointly on cross-server link bandwidth, heterogeneous GPU computing capability, GPU-CPU expert loading delay, instantaneous queueing backlog, and replica-level quantization quality loss. Existing distributed inference and MoE serving methods address these factors separately and do not provide a unified framework for online multi-server collaborative routing. In this paper, we propose HetRoute, a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference. HetRoute introduces a unified per-assignment cost model that explicitly captures four cost components: cross-server transmission, GPU-CPU offloading, GPU computation with queueing, and quantization-induced quality penalty. Guided by this model, the offline stage determines expert server placement, GPU-CPU residency, and replica precision through a routing-cost-coupled deployment algorithm, while the online stage routes the Top-k activated expert set as a whole by minimizing the bottleneck layer cost via exact enumeration or beam search. Theoretical analysis establishes fallback feasibility, a bound on the number of participating servers, per-layer optimality for small candidate domains, and online computational complexity. Trace-driven evaluation on three MoE models over a heterogeneous 10-server edge testbed shows that HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%, cuts cross-server traffic by up to 72.1%, and achieves 2.13x throughput improvement compared with representative baselines, while keeping quality degradation within the configured budget.

Related papers

• TrimMoE A communication aware and adaptive depth framework for distributed edge inference [44.83947237773358]
  Serving Mixture-of-Experts (MoE) large language models across distributed edge servers are bottlenecked by cross-server expert transmission.<n>We propose TrimMoE, which couples layer skipping and confidence-based early exit with substitute execution and server-expert selection.<n>On a heterogeneous 10-server testbed, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load.
  arXiv  Detail & Related papers  (2026-08-01T10:19:02Z)
• GeoRouteNet: Geometry-Enhanced Non-Autoregressive Neural Solver for the Traveling Salesman Problem [5.202950948929751]
  GeoRouteNet is a geometry-enhanced neural solver for Euclidean traveling salesman problem (TSP)<n>On the model side, GeoRouteNet incorporates centered node features, learnable radial distance basis functions, distance-aware graph attention, LayerNorm-SwiGLU feed-forward blocks, and cross-layer attentive residual encoder mixing.<n>On the training side, we design multi-candidate self-comparison reinforcement learning (MCS-RL), which samples multiple candidate tours per instance.
  arXiv  Detail & Related papers  (2026-06-22T02:31:22Z)
• INFRAMIND: Infrastructure-Aware Multi-Agent Orchestration [18.376319491516266]
  We introduce INFRAMIND, a framework that makes the entire multi-agent stack infrastructure-aware.<n>An infra-aware executor then observes per-model queue depths, cache utilization, and response latencies at each agent step to decide which model to call.<n>A budget-aware scheduler further reorders each model's queue so that urgent requests are served first.
  arXiv  Detail & Related papers  (2026-06-09T20:50:12Z)
• CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference [52.849509991178884]
  Large language models (LLMs) move from centralized clouds to mobile edge environments.<n> query-level routing between lightweight on-device models and stronger edge models provides a flexible mechanism to navigate this trade-off.<n>Existing routers are designed for centralized cloud settings and optimize token-level costs, failing to capture the dynamic latency and energy overheads in wireless edge deployments.
  arXiv  Detail & Related papers  (2026-05-12T11:50:15Z)
• ConsRoute:Consistency-Aware Adaptive Query Routing for Cloud-Edge-Device Large Language Models [7.869130026927]
  ConsRoute is a lightweight, semantic-aware, and adaptive routing framework for large language models.<n>We show that ConsRoute achieves near-cloud performance (>=95%) while reducing end-to-end latency and inference cost by nearly 40%.
  arXiv  Detail & Related papers  (2026-03-22T13:54:12Z)
• Efficient and Interpretable Multi-Agent LLM Routing via Ant Colony Optimization [58.59491516762626]
  We propose AMRO-S, an efficient and interpretable routing framework for Multi-Agent Systems (MAS)<n>AMRO-S models MAS routing as a semantic-conditioned path selection problem, enhancing routing performance through three key mechanisms.<n>Extensive experiments on five public benchmarks and high-concurrency stress tests demonstrate that AMRO-S consistently improves the quality--cost trade-off over strong routing baselines.
  arXiv  Detail & Related papers  (2026-03-13T12:26:05Z)
• RollArt: Scaling Agentic RL Training via Disaggregated Infrastructure [49.88201789074532]
  Agentic Reinforcement Learning (RL) enables Large Language Models (LLMs) to perform autonomous decision-making and long-term planning.<n>We present RollArc, a distributed system designed to maximize throughput for multi-task agentic RL on disaggregated infrastructure.
  arXiv  Detail & Related papers  (2025-12-27T11:14:23Z)
• ZeroLM: Data-Free Transformer Architecture Search for Language Models [54.83882149157548]
  Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity.<n>This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics.<n>Our evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark.
  arXiv  Detail & Related papers  (2025-03-24T13:11:22Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.