Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study

• URL: http://arxiv.org/abs/2608.18261v1
• Date: Tue, 18 Aug 2026 19:23:32 GMT
• Title: Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study
• Abstract summary: We quantify bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB)<n>We build llamamoe-trace, a zero-surgery router-telemetry tool, and measure routing on Q3-30B.<n>We pre-register training of 137M MoE language models with auxiliary locality and domain router losses.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.

Related papers

• Shared KV Caching for Replicated 27B Inference: Correctness Failures and Performance Boundaries [6.185334606321749]
  We study two single-GPU 27B vLLM replicas sharing a 256 GiB LMCache pool.<n>Six-turn synthetic sessions alternating replicas improve by approximately 35% and 45% at initial contexts of 32k and 128k.
  arXiv  Detail & Related papers  (2026-09-14T04:37:44Z)
• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• CacheRoute: Planned Prefix-Affinity Routing for Large-Scale LLM Serving [0.0]
  CacheRoute sustains 176 +/-11 QPS at a 3.5-s p99 SLO, 2.3x the strongest of five baselines.<n>On Llama-3.3-70B in fp8 across 60 H100 GPUs, CacheRoute sustains 176 +/-11 QPS at a 3.5-s p99 SLO, 2.3x the strongest of five baselines.
  arXiv  Detail & Related papers  (2026-08-20T06:12:47Z)
• Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy [0.0]
  We ask whether a Large Language Model can replace the static routing policy itself.<n>We find a capability threshold near 3B active parameters.<n>Above it, every model, regardless of architecture, saturates near an 88% reduction in client-perceived 5xx errors.
  arXiv  Detail & Related papers  (2026-08-11T06:15:20Z)
• Governing the KV Cache: Preventing Timing Side-Channel Leakage in Multi-Tenant LLM Inference [0.0]
  Key-value (KV) cache is the primary throughput optimization in modern large language model (LLM) inference.<n>Three published attacks exploit it, reaching up to 100% attack success rate against unprotected vLLM and SG.<n>We present KVGov, a governance layer addressing all three attack families' prefix-cache paths.
  arXiv  Detail & Related papers  (2026-08-10T07:47:57Z)
• When Does Trace-Driven Evaluation Mislead MoE Expert Caching? Replay Semantics, Workload Contamination, and Operating Regimes [3.471896896439183]
  Mixture-of-Experts (MoE) models have outgrown accelerator memory.<n>Policy that raised the hit rate would cut expert traffic per token.<n>We evaluate three evaluation axes that change conclusions, not just numbers.
  arXiv  Detail & Related papers  (2026-08-08T04:29:20Z)
• HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference [47.0879800334801]
  HetRoute is a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference.<n>HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%.
  arXiv  Detail & Related papers  (2026-08-01T10:30:01Z)
• Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabrics [1.081571058570587]
  Multi-head Latent Attention inverts arithmetic, compressing each token's key and value into one narrow vector.<n>We characterize cross-instance MLA attention on a real multi-node H100 cluster.
  arXiv  Detail & Related papers  (2026-05-31T23:53:24Z)
• Resource-Efficient Iterative LLM-Based NAS with Feedback Memory [49.44875022114861]
  Neural Architecture Search (NAS) automates network design, but conventional methods demand substantial computational resources.<n>We propose a closed-loop pipeline leveraging large language models (LLMs) to iteratively generate, evaluate, and refine convolutional neural network architectures.
  arXiv  Detail & Related papers  (2026-03-12T16:00:22Z)
• From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing [52.01745035243826]
  Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts.<n> conditional routing shifts the burden on inference memory, limiting the number of experts per device.<n>We present LASER, a plug-and-play, inference-time routing algorithm that balances load while preserving accuracy.
  arXiv  Detail & Related papers  (2025-09-29T16:29:17Z)
• A Simple Linear Patch Revives Layer-Pruned Large Language Models [58.056251480151104]
  Layer pruning has emerged as a widely used technique for compressing large language models (LLMs)<n>textscLinearPatch fuses two operations into one matrix multiply at the pruning interface.<n>The patch can be further refined with 5K unlabeled samples via memory-efficient offline distillation, pushing the retention to 95.16% within only 30 minutes on a single GPU.
  arXiv  Detail & Related papers  (2025-05-30T15:06:08Z)
• MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models [72.61076288351201]
  We propose Memory-efficient Offloaded Mini-sequence Inference (MOM)<n>MOM partitions critical layers into smaller "mini-sequences" and integrates seamlessly with KV cache offloading.<n>On Meta-Llama-3.2-8B, MOM extends the maximum context length from 155k to 455k tokens on a single A100 80GB GPU.
  arXiv  Detail & Related papers  (2025-04-16T23:15:09Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.