TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms

• URL: http://arxiv.org/abs/2501.02600v1
• Date: Sun, 05 Jan 2025 16:51:17 GMT
• Title: TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
• Abstract summary: The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in the cloud.<n>We propose TAPAS, a thermal-aware framework designed for LLM inference clusters in the cloud.<n>Our evaluation on a large GPU cluster demonstrates significant reductions in thermal and power throttling events, boosting system efficiency.
• Score: 9.36320423249322
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The rising demand for generative large language models (LLMs) poses challenges for thermal and power management in cloud datacenters. Traditional techniques often are inadequate for LLM inference due to the fine-grained, millisecond-scale execution phases, each with distinct performance, thermal, and power profiles. Additionally, LLM inference workloads are sensitive to various configuration parameters (e.g., model parallelism, size, and quantization) that involve trade-offs between performance, temperature, power, and output quality. Moreover, clouds often co-locate SaaS and IaaS workloads, each with different levels of visibility and flexibility. We propose TAPAS, a thermal- and power-aware framework designed for LLM inference clusters in the cloud. TAPAS enhances cooling and power oversubscription capabilities, reducing the total cost of ownership (TCO) while effectively handling emergencies (e.g., cooling and power failures). The system leverages historical temperature and power data, along with the adaptability of SaaS workloads, to: (1) efficiently place new GPU workload VMs within cooling and power constraints, (2) route LLM inference requests across SaaS VMs, and (3) reconfigure SaaS VMs to manage load spikes and emergency situations. Our evaluation on a large GPU cluster demonstrates significant reductions in thermal and power throttling events, boosting system efficiency.

Related papers

• LLM Inference in a Flash! [66.43896490006965]
  Large Language Models (LLMs) have shown impressive capabilities across a range of natural language processing tasks.<n>We present an end-to-end integer-only quantization approach to eliminate expensive floating-point computations.
  arXiv  Detail & Related papers  (2026-09-14T18:04:30Z)
• PELM: Power Efficient On-Device LLM Inference with Speculative Decoding and Dynamic Voltage Frequency Scaling [0.42015796373418257]
  Large Language Models (LLMs) can be deployed directly on mobile platforms at the edge.<n>LLMs have heavy computational requirements, which are difficult for resource-constrained mobile and edge platforms to fulfill.<n>We present PELM, a solution that augments traditional DVFS processor frequency tuning with two additional workload-specific knobs.
  arXiv  Detail & Related papers  (2026-09-09T03:27:51Z)
• EnerInfer: Energy-Aware On-Device LLM Inference [26.942672033524826]
  EnerInfer is a framework that jointly manages energy efficiency, throughput, and thermal comfort for LLM workloads.<n>EnerInfer improves energy efficiency by up to 65%, 12%, and 24% on phones, a laptop, and a development board, respectively.
  arXiv  Detail & Related papers  (2026-06-22T08:16:19Z)
• PALS: Power-Aware LLM Serving for Mixture-of-Experts Models [7.28734124763028]
  Large language model (LLM) inference has become a dominant workload in modern data centers.<n>We present a power-aware runtime for LLM serving, PALS, that treats GPU power caps as a first-class control knob.<n> PALS improves energy efficiency by up to 26.3%, reduces runtime violations by 4x to 7x under power constraints, and tracks dynamic power budgets.
  arXiv  Detail & Related papers  (2026-05-20T17:19:20Z)
• Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations [53.53864188270018]
  Large language models (LLMs) have advanced recommendation systems (RSs)<n>We propose MARC to explicitly control the modularity of LLMs.<n>MARC achieved a 2.82% eCPM lift in an online A/B test within a large-scale commercial search advertising scenario.
  arXiv  Detail & Related papers  (2026-04-20T12:08:58Z)
• Ge$^\ ext{2}$mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformer [84.8831358775386]
  Spiking Neural Networks (SNNs) offer superior energy efficiency over Artificial Neural Networks (ANNs)<n>Existing paradigms including ANN-SNN Conversion and Spatial-Temporal Backpropagation (STBP) suffer from inherent limitations.<n>We propose Ge$text2$mS-T, a novel architecture implementing grouped computation across temporal, spatial and network structure dimensions.
  arXiv  Detail & Related papers  (2026-04-10T02:58:46Z)
• Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use [4.513690948889834]
  Large Language Models (LLMs) are increasingly deployed in production, contributing towards shifting the burden in terms of computational resources and energy demands from training to inference.<n>We show how emphsystem-level design choices can lead to orders-of-magnitude differences in energy consumption for the same model.<n>Our findings motivate phase-aware energy profiling and system-level optimizations for greener AI services.
  arXiv  Detail & Related papers  (2026-01-29T22:16:25Z)
• RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs [48.94639777633359]
  We present RLBoost, a systematic solution for cost-efficient RL training that harvests preemptible GPU resources.<n> RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources.
  arXiv  Detail & Related papers  (2025-10-22T04:19:37Z)
• Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspective [6.51239603014107]
  Large Language Models (LLMs) have pushed training workloads beyond the limits of single-node analysis.<n>We present a comprehensive characterization of LLM training across diverse real-world workloads and hardware platforms.
  arXiv  Detail & Related papers  (2025-09-12T16:05:07Z)
• CSGO: Generalized Optimization for Cold Start in Wireless Collaborative Edge LLM Systems [62.24576366776727]
  We propose a latency-aware scheduling framework to minimize total inference latency.<n>We show that the proposed method significantly reduces cold-start latency compared to baseline strategies.
  arXiv  Detail & Related papers  (2025-08-15T07:49:22Z)
• DILEMMA: Joint LLM Quantization and Distributed LLM Inference Over Edge Computing Systems [1.14179290793997]
  This paper introduces DILEMMA, a novel framework addressing the challenges of deploying Large Language Models in Edge Computing systems. DILEMMA formulates an Linear Programming problem to minimize total delay while ensuring acceptable LLM performance levels. It achieves a quantization ratio of up to 12.75% while preserving model loss, highlighting its effectiveness in resource-constrained environments.
  arXiv  Detail & Related papers  (2025-03-03T16:16:33Z)
• Sparse Gradient Compression for Fine-Tuning Large Language Models [58.44973963468691]
  Fine-tuning large language models (LLMs) for downstream tasks has become increasingly crucial due to their widespread use and the growing availability of open-source models. High memory costs associated with fine-tuning remain a significant challenge, especially as models increase in size. We propose sparse compression gradient (SGC) to address these limitations.
  arXiv  Detail & Related papers  (2025-02-01T04:18:28Z)
• SLO-aware GPU Frequency Scaling for Energy Efficient LLM Inference Serving [6.010159688581912]
  We present textitthrottLL'eM, a framework that reduces energy consumption while meeting Service-Level Objectives. textitthrottLL'eM features mechanisms that project future KV cache usage and batch size. We show that the proposed ML model achieves $R2$ scores greater than 0.97 and miss-predicts performance by less than 1 iteration per second on average.
  arXiv  Detail & Related papers  (2024-08-05T09:07:06Z)
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency [7.073435885680335]
  We propose DynamoLLM, the first energy-management framework for generative large language models. At a service-level, DynamoLLM conserves 53% energy and 38% operational carbon emissions, and reduces 61% cost to the customer.
  arXiv  Detail & Related papers  (2024-08-01T17:40:45Z)
• Q-Sparse: All Large Language Models can be Fully Sparsely-Activated [93.45300714803429]
  We introduce Q-Sparse, a simple yet effective approach to training sparsely-activated large language models (LLMs) Q-Sparse enables full sparsity of activations in LLMs which can bring significant efficiency gains in inference. We also introduce Block Q-Sparse for batch training and inference.
  arXiv  Detail & Related papers  (2024-07-15T17:59:29Z)
• Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloads [0.2389598109913753]
  Training and using Large Language Models (LLMs) require large amounts of energy. This paper addresses the challenge of reducing energy consumption in data centers running LLMs. We propose a hybrid data center model that uses a cost-based scheduling framework to dynamically allocate tasks across hardware accelerators.
  arXiv  Detail & Related papers  (2024-04-25T11:24:08Z)
• LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization [9.517540904818986]
  This paper proposes adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. Experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference.
  arXiv  Detail & Related papers  (2024-03-02T08:40:07Z)
• SpotServe: Serving Generative Large Language Models on Preemptible Instances [64.18638174004151]
  SpotServe is the first distributed large language models serving system on preemptible instances. We show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances.
  arXiv  Detail & Related papers  (2023-11-27T06:31:17Z)
• FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs [57.12856172329322]
  We envision a decentralized system unlocking the potential vast untapped consumer-level GPU. This system faces critical challenges, including limited CPU and GPU memory, low network bandwidth, the variability of peer and device heterogeneity.
  arXiv  Detail & Related papers  (2023-09-03T13:27:56Z)
• POLCA: Power Oversubscription in LLM Cloud Providers [0.8299593158757622]
  Large language models (LLMs) are becoming increasingly power intensive. We show that there is a significant opportunity to oversubscribe power in LLM clusters. We propose POLCA, our framework for power oversubscription that is robust, reliable, and readily deployable for GPU clusters.
  arXiv  Detail & Related papers  (2023-08-24T16:32:34Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.