The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute

• URL: http://arxiv.org/abs/2505.14733v2
• Date: Sun, 09 Nov 2025 19:54:46 GMT
• Title: The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Compute
• Abstract summary: Scaling large language models (LLMs) has driven significant advancements, yet it faces diminishing returns and escalating energy demands.<n>This work explores how test-time compute can serve as an energy-efficient complement to conventional scaling strategies.
• Score: 5.83525198890183
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling large language models (LLMs) has driven significant advancements, yet it faces diminishing returns and escalating energy demands. This work explores how test-time compute (TTC) can serve as an energy-efficient complement to conventional scaling strategies by allocating additional computational resources at inference time rather than during training. Specifically, we investigate whether employing TTC can achieve superior accuracy-energy trade-offs compared to simply increasing model size. Our empirical analysis reveals that TTC surpasses traditional model scaling in accuracy/energy efficiency, with notable gains in tasks demanding complex reasoning rather than mere factual recall. Further, we identify a critical interaction between TTC performance and output sequence length, demonstrating that strategically adjusting compute resources at inference time according to query complexity can substantially enhance efficiency. Our findings advocate for TTC as a promising direction, enabling more sustainable, accurate, and adaptable deployment of future language models.

Related papers

• From Tokens to Watt-hours: Analytical Energy Estimation for LLM Inference on Modern GPUs [6.041312734387813]
  This report presents an analytically structured, empirically calibrated, GPU-level methodology for estimating large language model (LLM) inference energy on NVIDIA H100-class accelerators without direct measurement.<n>The proposed estimator combines parameter-scaled transformer FLOP accounting, memory-traffic factors, and hardware-specific energy coefficients for FP16/BF16 tensor-core computation and high-bandwidth-memory movement.<n>The methodology further decomposes total energy into compute, parameter-access, key-value-cache write, and attention-read components, allowing the scaling behavior with model size, context length, and generated-token count to
  arXiv  Detail & Related papers  (2026-07-29T07:50:55Z)
• Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference [1.522424334864671]
  Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs.<n>Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks.<n>These strategies can also introduce additional energy costs, creating an energy-accuracy trade-off.
  arXiv  Detail & Related papers  (2026-03-04T15:20:55Z)
• Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use [4.513690948889834]
  Large Language Models (LLMs) are increasingly deployed in production, contributing towards shifting the burden in terms of computational resources and energy demands from training to inference.<n>We show how emphsystem-level design choices can lead to orders-of-magnitude differences in energy consumption for the same model.<n>Our findings motivate phase-aware energy profiling and system-level optimizations for greener AI services.
  arXiv  Detail & Related papers  (2026-01-29T22:16:25Z)
• Knowledge Distillation for Temporal Knowledge Graph Reasoning with Large Language Models [8.46502493796591]
  Reasoning over temporal knowledge graphs (TKGs) is fundamental to improving the efficiency and reliability of intelligent decision-making systems.<n>Existing TKG reasoning models typically rely on large parameter sizes and intensive computation.<n>We propose a distillation framework specifically tailored for temporal knowledge graph reasoning.
  arXiv  Detail & Related papers  (2026-01-01T04:38:00Z)
• Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression [53.39128997308138]
  We introduce information capacity, a measure of model efficiency based on text compression performance.<n> Empirical evaluations on mainstream open-source models show that models of varying sizes within a series exhibit consistent information capacity.<n>A distinctive feature of information capacity is that it incorporates tokenizer efficiency, which affects both input and output token counts.
  arXiv  Detail & Related papers  (2025-11-11T10:07:32Z)
• Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Models [64.02612380298228]
  Recent studies have explored the idea of continuing to train a model at test-time for a given task, known as test-time training (TTT)<n>We propose a model in which TTT achieves a substantially smaller in-distribution test error than global training.<n>We empirically validate our model's key assumptions by training a sparse autoencoder on ImageNet.
  arXiv  Detail & Related papers  (2025-09-29T09:24:52Z)
• Comparing energy consumption and accuracy in text classification inference [0.9208007322096533]
  This study systematically evaluates the trade-offs between model accuracy and energy consumption in text classification inference.<n>The best-performing model in terms of accuracy can also be energy-efficient, while larger LLMs tend to consume significantly more energy with lower classification accuracy.
  arXiv  Detail & Related papers  (2025-08-19T18:00:08Z)
• Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs [45.83245433138508]
  Large language models (LLMs) have rapidly progressed into general-purpose agents capable of solving a broad spectrum of tasks.<n>They apply fixed inference-time compute regardless of task complexity, often overthinking simple problems while underthinking hard ones.<n>This survey presents a comprehensive review of efficient test-time compute strategies, which aim to improve the computational efficiency of LLM reasoning.
  arXiv  Detail & Related papers  (2025-07-02T18:27:42Z)
• Lightweight Task-Oriented Semantic Communication Empowered by Large-Scale AI Models [66.57755931421285]
  Large-scale artificial intelligence (LAI) models pose significant challenges for real-time communication scenarios.<n>This paper proposes utilizing knowledge distillation (KD) techniques to extract and condense knowledge from LAI models.<n>We propose a fast distillation method featuring a pre-stored compression mechanism that eliminates the need for repetitive inference.
  arXiv  Detail & Related papers  (2025-06-16T08:42:16Z)
• Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models [7.2703757624760526]
  Large reasoning models (LRMs) have exhibited the capacity of enhancing reasoning performance via internal test-time scaling.<n>As we push these scaling boundaries, understanding the practical limits and achieving optimal resource allocation becomes a critical challenge.<n>In this paper, we investigate the scaling plateau of test-time scaling and introduce the Test-Time Scaling Performance Model (TTSPM)
  arXiv  Detail & Related papers  (2025-05-26T20:58:45Z)
• DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs [70.91804882618243]
  This paper proposes DSMoE, a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks.<n>We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge.<n>Experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches.
  arXiv  Detail & Related papers  (2025-02-18T02:37:26Z)
• When More is Less: Understanding Chain-of-Thought Length in LLMs [53.77747102201451]
  Chain-of-thought (CoT) reasoning enhances the multi-step reasoning capabilities of large language models (LLMs)<n>However, for most models and tasks, does an increase in CoT length consistently lead to improved reasoning accuracy?<n>In this paper, we observe a nuanced relationship: as the number of reasoning steps increases, performance initially improves but eventually decreases.
  arXiv  Detail & Related papers  (2025-02-11T05:28:59Z)
• An Inquiry into Datacenter TCO for LLM Inference with FP8 [7.910301381209274]
  We analyze the computational characteristics and constraints of large language models (LLMs) inference from a TCO perspective.<n>We present a generalizable framework that enables CSPs to compare and select AI accelerators according to diverse operational requirements.
  arXiv  Detail & Related papers  (2025-02-03T05:26:22Z)
• Context-Preserving Tensorial Reconfiguration in Large Language Model Training [0.0]
  Context-Preservingial Reconfiguration (CPTR) enables dynamic complexity of weight tensors through structured factorization and adaptive contraction.<n> Empirical evaluations demonstrate that CPTR improves coherence retention across extended sequences.<n>Performance comparisons reveal that CPTR-enhanced models exhibit greater computational efficiency and reduced memory consumption.
  arXiv  Detail & Related papers  (2025-02-01T00:55:19Z)
• Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity [89.81738321188391]
  This study investigates the relationship between task complexity and optimal sparsity in SMoE models.<n>We show that the optimal sparsity lies between minimal activation (1-2 experts) and full activation, with the exact number scaling proportionally to task complexity.
  arXiv  Detail & Related papers  (2024-10-17T18:40:48Z)
• From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency [17.612497960364916]
  Chain-of-thought (CoT) significantly enhances the reasoning performance of large language models (LLM)<n>We demonstrate that CoT can substantially improve sample efficiency even when representation power is sufficient.<n>We show that CoT simplifies the learning process by introducing sparse dependencies among input tokens, and leads to a sparse and interpretable attention.
  arXiv  Detail & Related papers  (2024-10-07T19:45:09Z)
• TEA: Test-time Energy Adaptation [67.4574269851666]
  Test-time adaptation (TTA) aims to improve model generalizability when test data diverges from training distribution. We propose a novel energy-based perspective, enhancing the model's perception of target data distributions.
  arXiv  Detail & Related papers  (2023-11-24T10:49:49Z)
• Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models [109.06052781040916]
  We introduce a technique to enhance the inference efficiency of parameter-shared language models. We also propose a simple pre-training technique that leads to fully or partially shared models. Results demonstrate the effectiveness of our methods on both autoregressive and autoencoding PLMs.
  arXiv  Detail & Related papers  (2023-10-19T15:13:58Z)
• Energy-Efficient and Real-Time Sensing for Federated Continual Learning via Sample-Driven Control [21.871879862642235]
  Real-Time Sensing (RTS) systems must continuously acquire, update, integrate, and apply knowledge to adapt to real-world dynamics.<n>We investigate how the data distribution shift from ideal to practical RTS scenarios affects Artificial Intelligence (AI) model performance.<n>We develop a novel Sample-driven Control for Federated Continual Learning (SCFL) technique, specifically designed for mobile edge networks with RTS capabilities.
  arXiv  Detail & Related papers  (2023-10-11T13:50:28Z)
• In Search of Deep Learning Architectures for Load Forecasting: A Comparative Analysis and the Impact of the Covid-19 Pandemic on Model Performance [0.0]
  Short-term load forecasting (STLF) is crucial to the optimization of their reliability, emissions, and costs. This work conducts a comparative study of Deep Learning (DL) architectures, with respect to forecasting accuracy and training sustainability. The case study focuses on day-ahead forecasts for the Portuguese national 15-minute resolution net load time series.
  arXiv  Detail & Related papers  (2023-02-25T10:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.