Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances

• URL: http://arxiv.org/abs/2609.12550v1
• Date: Fri, 11 Sep 2026 07:57:25 GMT
• Title: Quality-Constrained Routing over a Fixed Pool of Quantized Mixture-of-Experts Instances
• Abstract summary: Quantized Mixture-of-Experts (MoE) services can hold several pre-materialized instances of one base model.<n>Because instance materialization and replica counts consume memory, we treat them as upstream provisioning decisions.<n>We route each request to maximize modeled throughput under a class-level expected quality-degradation budget.
• Score: 5.053749663617978
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantized Mixture-of-Experts (MoE) services can hold several pre-materialized instances of one base model, but quantization damage varies sharply across requests and bitwidths. Because instance materialization and replica counts consume memory and require slow reconfiguration, we treat them as upstream provisioning decisions and study routing within a fixed resident pool. Within this fixed-pool boundary, we route each request to maximize modeled throughput under a class-level expected quality-degradation budget and measured instance capacities. To predict this request-specific risk, we introduce FWP (Fragility-Weighted Perplexity), computed from prompt tokens on a reference-instance prefill and calibrated to candidate-instance degradation. Underlying FWP is an exact two-expert affinity--fragility decomposition and a conditional multi-layer top-$k$ expansion whose bias, interaction, route-change, separability, and higher-order terms remain explicit. Using these calibrated risks, a window-level linear program yields a signed reduced-reward score that is KKT-consistent with the LP optimum under optimal prices and primal-feasible tie allocation. On 88 extended Qwen prompts, complete W2, W3, and W4 instances quantizing all 6,144 expert blocks incur mean $Δ$NLL of $0.9437$, $0.1832$, and $0.0513$. Under the same population and $τ=0.1513$, FWP allocation reaches a $1.284\times$ offline model-based multiplier versus $1.253\times$ for request-agnostic mixing and $1.000\times$ for static W4, an incremental $2.5\%$ relative FWP gain.

Related papers

• SoftWater: Class-Aware Rate Allocation for Softmax Quantization [5.213778368155993]
  Post-training quantization pipelines routinely leave the softmax output layer in high precision.<n>In small LLMs with modern vocabularies, the head holds 15--30% of all parameters, so a nominal 2-bit'' model with an fp16 head can store several times as many bits per weight.<n>We pose softmax-layer quantization as a rate-distortion problem under the KL divergence between the original and quantized output distributions.
  arXiv  Detail & Related papers  (2026-08-12T13:06:50Z)
• SSTQ:Privacy-Preserving Vector Quantization via Subsampled Stochastic TurboQuant [79.24089819400126]
  Subsampled TurboQuant (SSTQ) is a framework that combines overcomplete equal-width tight frames, coordinate subsampling, and privacy-aware quantization.<n>SSTQ achieves mean squared error scaling while using only $lceil log N il + b$ bits per client, where $N = (d)$ is the frame size.<n>We also derive a surrogate privacy-aware codebook objective that reduces the codebook-dependent MSE scaling from $O(4b)$ to $O(2b)$.
  arXiv  Detail & Related papers  (2026-08-05T17:51:25Z)
• Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit [0.30586855806896046]
  We adapt the classical paired sample-size calculation to quantization benchmarks.<n>The bound turns "how reliable is my quantization claim?" into a one-line budget.<n>We complement the bound with a five-line pre-registration template.
  arXiv  Detail & Related papers  (2026-05-25T07:13:35Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• AESOP: Adversarial Execution-path Selection to Overload Deep Learning Pipelines [6.773252452930085]
  Modern machine learning deployments increasingly compose specialized models into dynamic inference pipelines.<n>We show that this structure creates an efficiency-attack surface that existing methods targeting single models cannot exploit.<n>We present AESOP, a framework combining vulnerability-guided path ranking with adaptive loss weighting.
  arXiv  Detail & Related papers  (2026-05-09T09:16:49Z)
• Dual-Pool Token-Budget Routing for Cost-Efficient and Reliable LLM Serving [9.457255218406333]
  Production vLLM fleets typically provision each instance for the worst-case context length.<n>In practice, 80-95% of requests are short, yet are served under configurations optimized for long contexts.<n>We propose dual-pool token-budget routing, a lightweight dispatch mechanism that partitions a homogeneous fleet into two specialized pools.
  arXiv  Detail & Related papers  (2026-04-09T10:47:20Z)
• Topological Characterization of Churn Flow and Unsupervised Correction to the Wu Flow-Regime Map in Small-Diameter Vertical Pipes [0.0]
  We introduce the first topology-based characterization using Characteristic Surfaces (ECS)<n>This work provides the first mathematical definition of churn flow and demonstrates that unsupervised topological descriptors can challenge and correct mechanistic models.
  arXiv  Detail & Related papers  (2026-04-07T17:59:15Z)
• Constrained Pricing under Finite Mixtures of Logit [12.60044184194183]
  We study the constrained pricing problem under multinomial and mixed logit demand models.<n>We show that constrained pricing under finite mixtures of logit admits a PTAS when the number of customer segments is bounded.
  arXiv  Detail & Related papers  (2026-02-08T20:48:50Z)
• Multiple-play Stochastic Bandits with Prioritized Arm Capacity Sharing [52.124267908936396]
  The model is composed of $M$ arms and $K$ plays.<n>Each arm has a number of capacities, and each unit of capacity is associated with a reward function.<n>When multiple plays compete for the arm capacity, the arm capacity is allocated in a larger priority weight first manner.
  arXiv  Detail & Related papers  (2025-12-25T11:19:09Z)
• Probabilistically Tightened Linear Relaxation-based Perturbation Analysis for Neural Network Verification [83.25968588249776]
  We present a novel framework that combines over-approximation techniques from LiRPA-based approaches with a sampling-based method to compute tight intermediate reachable sets.<n>With negligible computational overhead, $textttPT-LiRPA$ exploiting the estimated reachable sets, significantly tightens the lower and upper linear bounds of a neural network's output.
  arXiv  Detail & Related papers  (2025-07-07T18:45:53Z)
• RePO: Understanding Preference Learning Through ReLU-Based Optimization [66.098833436503]
  We propose ReLU-based Preference Optimization (RePO), a streamlined algorithm that eliminates $beta$ via two advances.<n>RePO is characterized as SimPO's limiting case ($beta to infty$), where the logistic weighting collapses to binary thresholding.<n> Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models.
  arXiv  Detail & Related papers  (2025-03-10T15:11:07Z)
• On Optimal Caching and Model Multiplexing for Large Model Inference [66.50550915522551]
  Large Language Models (LLMs) and other large foundation models have achieved noteworthy success, but their size exacerbates existing resource consumption and latency challenges. We study two approaches for mitigating these challenges: employing a cache to store previous queries and learning a model multiplexer to choose from an ensemble of models for query processing.
  arXiv  Detail & Related papers  (2023-06-03T05:01:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.