Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving

• URL: http://arxiv.org/abs/2609.11133v1
• Date: Thu, 10 Sep 2026 06:23:37 GMT
• Title: Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving
• Abstract summary: We deploy NVIDIA's Max-Q inference profile on a disaggregated B200 system.<n>On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e.
• Score: 1.249668212599889
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Datacenter GPU power is the binding constraint on LLM serving capacity, and production serving has shifted to prefill/decode (PD) disaggregation. Deploying NVIDIA's Max-Q inference profile on a disaggregated B200 system, we found its realized gain modest (+8.6% tokens/J), model-dependent, and carrying a mean end-to-end latency cost (+5.2%) that throughput-only evaluation does not surface; the profile also applies one setting to prefill and decode GPUs that operate in opposite hardware regimes. We hypothesize that the optimal power setting is a property of the deployed (model, quantization, engine, hardware) combination rather than of the GPU class, that each lane warrants its own profile, and that converting SLO headroom into energy safely requires latency-gated calibration under a runtime SLO guard rather than a fixed recipe. We present a phase-decoupled, model-calibrated controller: the prefill lane runs under an SM-clock window whose floor is a latency guarantee by construction, and the decode lane under a power cap placed by automatic calibration just above a measured throughput/latency cliff. Because a disaggregated decode lane draws flat, memory-bound power, the cap binds continuously, the reactive-overshoot weakness that led POLCA to reject capping is absent, and the GPU's own power manager retains throughput under the cap. On an 8x B200 node serving Qwen3-Coder-480B (FP8) under agentic load, our balanced mode delivers +20.4% tokens/J at +3.5% mean e2e versus +8.6% at +5.2% for Max-Q, a Pareto improvement on both axes. On Qwen3-235B-A22B (NVFP4) every operating mode meets the ITL-p99 SLO in every repetition; both vendor profiles miss it. A decode-actuator A/B shows the calibrated cap beats static clock locks, and a three-day sustained run saves 32.3% of a lane pair's electricity. Both models are MoE; a dense model recovers roughly 5x less, so we scope our claims to MoE serving.

Related papers

• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy [0.0]
  We ask whether a Large Language Model can replace the static routing policy itself.<n>We find a capability threshold near 3B active parameters.<n>Above it, every model, regardless of architecture, saturates near an 88% reduction in client-perceived 5xx errors.
  arXiv  Detail & Related papers  (2026-08-11T06:15:20Z)
• LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving [12.697860184583483]
  Co-batching improves throughput but obscures per-tenant usage and limits control.<n>We present LLMVisor, a roofline-guided latency model that captures the memory-bound and compute-bound phases.<n>Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x.
  arXiv  Detail & Related papers  (2026-08-09T00:28:05Z)
• Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening [6.384763560610076]
  We report operational experience full-fine-tuning a 32.76B- parameter dense model (Qwen3-32B) on 16 x NVIDIA B300 (two nodes, FSDP / ZeRO-3)<n>Our contribution is the integrated field experience and a set of calibrated measurements on new hardware.<n>We position our instantiation against PyTorch's documented Join / equalize-to-minimum practice.
  arXiv  Detail & Related papers  (2026-08-06T12:11:25Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Think Before You Grid-Search: Floor-First Triage for LLM Serving [0.0]
  LLM serving optimization typically benchmarks many configurations and reaches for heavy profilers when latency targets are missed.<n>We argue for the reverse discipline: estimation is the analytical layer of profiling.
  arXiv  Detail & Related papers  (2026-07-07T06:11:54Z)
• Reliability-Asymmetric Spacecraft Autonomy: Co-Designing a Capable Learned GNC Stack with a Verified, Adaptation-Aware Runtime Shield [2.28438857884398]
  Rule-based autonomy is certifiable but brittle, while learned autonomy is capable but hard to verify.<n>We present AMPLE-GNC, a three-tier guidance, navigation, and control stack.<n>Fine-tuning a pretrained 360M model with grammar-constrained decoding gives a hard output-validity guarantee.<n>We show that a latching safe-hold shield can suppress even a capable controller.
  arXiv  Detail & Related papers  (2026-06-24T03:55:21Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• OverFill: Two-Stage Models for Efficient Language Model Decoding [68.68408155020568]
  Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs.<n>We propose OverFill, which decouples prefill and decode stages to optimize accuracy-efficiency tradeoffs.<n>Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average.
  arXiv  Detail & Related papers  (2025-08-11T20:07:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.