The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment

• URL: http://arxiv.org/abs/2605.23918v1
• Date: Wed, 15 Apr 2026 09:01:24 GMT
• Title: The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment
• Abstract summary: AI industry keeps models loaded in memory around the clock to avoid cold-start latency.<n>Yet the structure of this cost has never been empirically decomposed.<n>We present the first cross-architecture measurement of idle GPU power as a function of VRAM allocation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The AI inference industry keeps models loaded in GPU memory around the clock to avoid cold-start latency, implicitly treating idle power as a fixed cost of readiness. Yet the structure of this cost has never been empirically decomposed - and never across GPU architectures. We present the first cross-architecture measurement of idle GPU power as a function of VRAM allocation, combining 18 days of production telemetry (335,267 samples, 14 H100 GPUs) with controlled dose-response experiments on three GPU architectures spanning three memory technologies: NVIDIA H100 (HBM3, 80 GB), A100 (HBM2e, 80 GB), and L40S (GDDR6, 48 GB). We observe that idle power is piecewise constant on all three architectures: the CUDA context forces a discrete DVFS transition consuming +26-66 W over bare idle (26-50 W on HBM architectures, 66 W on GDDR6), while the marginal VRAM effect is bounded below measurement relevance ($|β| < 0.02$ W/GB) on every device tested. The CUDA context accounts for >98% of the parking tax regardless of memory technology. We validate this finding with a real HuggingFace model (Qwen2.5-7B) on all three architectures, confirming <0.5 W difference from empty tensors on every device, and capture cold-start power profiles during model loading. We derive a cold-start breakeven model showing energy-optimal behavior depends on request arrival rate and loading latency - not model size - with breakeven intervals of 1-5 minutes. Our results identify a constraint consistent across all tested architectures: idle-with-context power is determined by DVFS state, not memory occupancy.

Related papers

• FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention [77.12062766962815]
  Lookahead Sparse Attention (LSA) is powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture.<n>We instantiate this architecture via a backbone-free decoupled training strategy.<n>We show that FM-DS-V4 compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline.
  arXiv  Detail & Related papers  (2026-06-08T06:25:54Z)
• Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX [2.6853734738584047]
  This paper presents a benchmark study comparing standard non-confidential execution with confidential computing mode on a single NVIDIA H100 80GB GPU.<n>The results suggest that confidential GPU inference can retain usable throughput under load, but capacity planning must account for both the steady throughput penalty and the earlier saturation behavior observed for larger models.
  arXiv  Detail & Related papers  (2026-05-20T22:52:12Z)
• ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning [58.54940026861599]
  textscChunkFT is a memory-efficient fine-tuning framework.<n>textscChunkFT enables gradient computation for arbitrary sub-tensors without modifying the network architecture.<n>textscChunkFT consistently outperforms existing memory-efficient baselines.
  arXiv  Detail & Related papers  (2026-05-20T13:44:44Z)
• OOM-Free Alpamayo via CPU-GPU Memory Swapping for Vision-Language-Action Models [1.1011268090482575]
  Vision-Language-Action (VLA) models for autonomous driving unify perception, reasoning, and control in a single neural network.<n>We present a framework, which enables memory-efficient VLA inference on VRAM-constrained GPU.<n>Our work achieves up to 3.55x speedup over Accelerate offloading while maintaining full BF16 precision.
  arXiv  Detail & Related papers  (2026-05-12T07:37:10Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• An Efficient Heterogeneous Co-Design for Fine-Tuning on a Single GPU [10.811837575220814]
  Fine-tuning Large Language Models (LLMs) has become essential for domain adaptation, but its memory-intensive property exceeds the capabilities of most GPU.<n>We present SlideFormer, a novel system designed for single-GPU environments.<n>A lightweight asynchronous engine that treats the GPU as a sliding window and overlaps GPU with CPU updates and multi-tier I/O.
  arXiv  Detail & Related papers  (2026-03-17T12:05:17Z)
• Light-Weight Benchmarks Reveal the Hidden Hardware Cost of Zero-Shot Tabular Foundation Models [0.0]
  We present a benchmark that reports test accuracy together with wall-clock latency, peak CPU RAM, and peak GPU VRAM on four public datasets.<n>Two open FMs (TabPFN-1.0 and TabICL-base) are compared against tuned XGBoost, LightGBM, and Random Forest baselines on a single NVIDIA T4 GPU.<n>The tree ensembles equal or surpass FM accuracy on three datasets while completing full-test batches in = 0.40 s and = 150 MB RAM, using zero VRAM.
  arXiv  Detail & Related papers  (2025-11-30T13:17:08Z)
• Minute-Long Videos with Dual Parallelisms [57.22737565366549]
  Diffusion Transformer (DiT)-based video diffusion models generate high-quality videos at scale but incur prohibitive processing latency and memory costs for long videos.<n>We propose a novel distributed inference strategy, termed DualParal.<n>Instead of generating an entire video on a single GPU, we parallelize both temporal frames and model layers across GPUs.
  arXiv  Detail & Related papers  (2025-05-27T11:55:22Z)
• HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading [79.38548165722229]
  HEADINFER offloads the KV cache to CPU RAM while avoiding the need to fully store the KV cache for any transformer layer on the GPU.<n>We demonstrate HEADINFER maintains computational efficiency while significantly reducing memory footprint.
  arXiv  Detail & Related papers  (2025-02-18T06:26:05Z)
• Deep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading [2.8231000588510757]
  Transformers and large language models(LLMs) have seen rapid adoption in all domains. Training of transformers is very expensive and often hits a memory wall'' We propose a novel technique to split the LLM into subgroups, whose update phase is scheduled on either the CPU or the GPU.
  arXiv  Detail & Related papers  (2024-10-26T00:43:59Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.