TrimMoE A communication aware and adaptive depth framework for distributed edge inference

• URL: http://arxiv.org/abs/2608.00573v1
• Date: Sat, 01 Aug 2026 10:19:02 GMT
• Title: TrimMoE A communication aware and adaptive depth framework for distributed edge inference
• Abstract summary: Serving Mixture-of-Experts (MoE) large language models across distributed edge servers are bottlenecked by cross-server expert transmission.<n>We propose TrimMoE, which couples layer skipping and confidence-based early exit with substitute execution and server-expert selection.<n>On a heterogeneous 10-server testbed, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load.
• Score: 44.83947237773358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Serving Mixture-of-Experts (MoE) large language models across distributed edge servers is bottlenecked by the cross-server expert transmission. The existing approaches mainly focus on how to reach a remote expert faster. However, in this paper, we instead consider whether a given layer, and the layers after it, need to be executed at all. To this end, a communication-aware adaptive-depth framework is proposed in this paper, termed TrimMoE, which couples layer skipping and confidence-based early exit with substitute execution and server-expert selection under a unified quality budget. Specifically, in the offline stage, TrimMoE freezes the backbone, trains the lightweight per-layer exit heads, calibrates the per-layer importance thresholds, and allocates the expert replicas by a skip/exit-aware redundancy benefit. In the online stage, a transition-aware look-ahead anticipates the token movement, so that the depth reduction targets the costliest transmissions, and besides, two feedback rules adapt the delay-quality weights and the exit threshold. Moreover, we prove that the substitution-and-skipping proxy degradation never exceeds the configured budget, and that the early exit is admitted only under a calibrated confidence gate. On a heterogeneous 10-server testbed with Switch-Base-8E, Qwen-MoE-A2.7B, and Mixtral-8x7B, TrimMoE reduces the average latency by up to 62.8%, lowers the cross-server traffic and the remote-execution ratio, and sustains high throughput under load, while keeping the task-quality degradation within a 2% bound.

Related papers

• When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models [14.959514582161956]
  Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models.<n>We observe that this assumption breaks down under over-dispersed routing, in which tokens are distributed nearly uniformly across experts.<n>We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected.
  arXiv  Detail & Related papers  (2026-09-03T20:21:20Z)
• HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference [47.0879800334801]
  HetRoute is a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference.<n>HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%.
  arXiv  Detail & Related papers  (2026-08-01T10:30:01Z)
• OrderMoE: An expert similarity driven distributed edge MoE inference [46.66948323823587]
  This paper introduces a similarity-aware expert allocation and distributed deployment framework, dubbed OrderMoE.<n>It aims to accelerate edge MoE inference while balancing inference latency, communication overhead, server workload, and inference quality.<n> Experimental results show that OrderMoE significantly reduces average latency, tail latency, cross-server traffic, and remote expert invocation ratio.
  arXiv  Detail & Related papers  (2026-07-19T09:20:21Z)
• DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation [69.8840101036735]
  Speculative decoding accelerates Large Language Model (LLM) inference.<n>Recent parallel drafters efficiently propose long token sequences in a single forward pass, but suffer from rapid acceptance decay due to a lack of inter-token dependencies.<n>We introduce DSpark, a speculative decoding framework that unifies high- throughput parallel generation with adaptive, load-aware verification.
  arXiv  Detail & Related papers  (2026-07-06T14:28:06Z)
• WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference [56.297697169678095]
  WISV (Wireless-Informed Semantic Verification) is a novel distributed speculative decoding framework.<n>We show that WISV achieves up to a 60.8% increase in accepted length, a 37.3% reduction in interaction rounds, and a 31.4% improvement in end-to-end latency.<n>We validate WISV on a hardware testbed comprising an NVIDIA Jetson AGX Orin and an A40-equipped server.
  arXiv  Detail & Related papers  (2026-04-20T01:29:56Z)
• Joint Routing and Model Pruning for Decentralized Federated Learning in Bandwidth-Constrained Multi-Hop Wireless Networks [16.79625047908881]
  Decentralized federated learning (D-FL) enables privacy-preserving training without a central server.<n>We propose a joint routing-and-pruning framework that optimize routing paths and pruning rates to maintain communication latency within prescribed limits.<n>We show that, compared with unpruned systems, the proposed framework reduces average transmission latency by 27.8% and improves testing accuracy by approximately 12%.
  arXiv  Detail & Related papers  (2026-03-16T12:25:27Z)
• Distributed Training under Packet Loss [8.613477072763404]
  Leveraging unreliable connections will reduce latency but may sacrifice model accuracy and convergence once packets are dropped.<n>We introduce a principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss.<n>This work bridges the gap between communication-efficient protocols and the accuracy and guarantees demanded by modern large-model training.
  arXiv  Detail & Related papers  (2025-07-02T11:07:20Z)
• Geminet: Learning the Duality-based Iterative Process for Lightweight Traffic Engineering in Changing Topologies [53.38648279089736]
  Geminet is a lightweight and scalable ML-based TE framework that can handle changing topologies.<n>Its neural network size is only 0.04% to 7% of existing schemes.<n>When trained on large-scale topologies, Geminet consumes under 10 GiB of memory, more than eight times less than the 80-plus GiB required by HARP.
  arXiv  Detail & Related papers  (2025-06-30T09:09:50Z)
• Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks [44.37047471448793]
  In this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL) We propose an innovative PSL framework, namely, efficient parallel split learning (EPSL) to accelerate model training. We show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy.
  arXiv  Detail & Related papers  (2023-03-26T16:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.