EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Serving

• URL: http://arxiv.org/abs/2605.10556v2
• Date: Wed, 13 May 2026 14:15:33 GMT
• Title: EnergyLens: Interpretable Closed-Form Energy Models for Multimodal LLM Inference Serving
• Abstract summary: Existing approaches either treat latency as an energy proxy or rely on data-hungry black-box surrogates.<n>We present EnergyLens, which uses symbolic regression as a structure-discovery tool over profiling data.<n>Unlike black-box surrogates, EnergyLens decouples tensor and pipeline parallelism contributions and separates prefill from decode energy.
• Score: 2.7498981662768536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models span dense, mixture-of-experts, and state-space architectures and are deployed on heterogeneous accelerators under increasingly diverse multimodal workloads, optimising inference energy has become as critical as optimizing latency and throughput. Existing approaches either treat latency as an energy proxy or rely on data-hungry black-box surrogates. Both fail under varying parallelism strategies: latency and energy optima diverge in over 20% of configurations we tested, and black-box surrogates require hundreds of profiling samples to generalize across model families and hardware. We present EnergyLens, which uses symbolic regression as a structure-discovery tool over profiling data to derive a single twelve-parameter closed-form energy model expressed in terms of system properties such as degree of parallelism, batch size, and sequence length. Unlike black-box surrogates, EnergyLens decouples tensor and pipeline parallelism contributions and separates prefill from decode energy, making its predictions physically interpretable and actionable. Fitted from as few as 50 profiling measurements, EnergyLens achieves 88.2% Top-1 configuration selection accuracy across many evaluation scenarios compared to 60.9% for the closest prior analytical baseline, matches the predictive accuracy of ensemble ML methods with 10x fewer profiling samples, and extrapolates reliably to unseen batch sizes and hardware platforms without structural modification, making it a practical, interpretable tool for energy-optimal LLM deployment.

Related papers

• Hyperparameter Scaling Laws Across MoE Sparsity [25.608085561102566]
  Mixture-of-Experts (MoE) models expand model capacity without a proportional increase in training compute.<n>We show that conventional hyperparameter scaling laws are insufficient for ultra-sparse MoEs.<n>We show that the optimal learning rate and batch size vary with activation ratio, and these shifts cannot be explained by either total or activated parameter count alone.
  arXiv  Detail & Related papers  (2026-09-08T12:57:20Z)
• A Defense of the Quadratic Model [71.57026347416729]
  We stress test the simplest possible model of optimization -- the quadratic model.<n>We show that it can be predictive surprisingly in an LLM setting with 150M parameters and 3B training tokens.
  arXiv  Detail & Related papers  (2026-07-23T18:02:03Z)
• EnergyLens: Predictive Energy-Aware Exploration for Multi-GPU LLM Inference Optimization [5.093812479614167]
  EnergyLens is an end-to-end framework for energy-aware large language model (LLM) inference optimization.<n>It captures LLM specifications including fusion, parallelism, and compute-communication overlap, combined with load-imbalance-aware MoE modeling.<n>We validate EnergyLens on Llama3 and Qwen3-MoE across tensor-parallel and expert-parallel configurations.<n>Our energy-driven exploration reveals up to 1.47x and 52.9x energy variation across configurations in prefill and decode efficiency and motivates distributed serving.
  arXiv  Detail & Related papers  (2026-05-14T01:37:26Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters [0.3186130813218338]
  Large Language Models (LLMs) are being increasingly used across a wide range of tasks.<n>Their substantial computational demands raise concerns about the energy efficiency and sustainability of both training and inference.<n>Recent research has explored optimization techniques and analyzed how configuration choices influence energy consumption.
  arXiv  Detail & Related papers  (2026-02-06T16:18:22Z)
• TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices [0.0]
  This paper investigates the effectiveness of small language models (SLMs) for agentic tasks (function/tool/API calling)<n>We describe parameter-driven optimization strategies that include supervised fine-tuning (SFT), parameter-efficient fine-tuning (PEFT), reinforcement learning (RL), and hybrid methods.<n>Our results demonstrate clear accuracy differences across model scales where medium-sized models (1-3B parameters) significantly outperform ultra-compact models (1B parameters)<n>This study highlights the importance of hybrid optimization strategies that enable small language models to deliver accurate, efficient, and stable agentic AI on edge devices.
  arXiv  Detail & Related papers  (2025-11-27T06:09:54Z)
• EPO: Diverse and Realistic Protein Ensemble Generation via Energy Preference Optimization [14.859985641146672]
  This paper presents Energy Preference Optimization (EPO), an online refinement that turns a pretrained protein ensemble generator into an energy-aware sampler.<n>On Tetrapeptides, ATLAS, and Fast-Folding benchmarks, EPO successfully generates diverse and physically realistic ensembles.
  arXiv  Detail & Related papers  (2025-11-13T10:25:50Z)
• EconProver: Towards More Economical Test-Time Scaling for Automated Theorem Proving [64.15371139980802]
  Large Language Models (LLMs) have recently advanced the field of Automated Theorem Proving (ATP)<n>We show that different test-time scaling strategies for ATP models introduce significant computational overhead for inference.<n>We propose two complementary methods that can be integrated into a unified EconRL pipeline for amplified benefits.
  arXiv  Detail & Related papers  (2025-09-16T03:00:13Z)
• Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining [59.369484219304866]
  We conduct an unprecedented empirical investigation training over 3,700 Large Language Models (LLMs) from scratch across 100 trillion tokens.<n>We establish a universal Scaling Law for hyperparameter optimization in LLM Pre-training, called Step Law.<n>Our estimated optima deviates from the global best performance found via exhaustive search by merely 0.094% on the test set.
  arXiv  Detail & Related papers  (2025-03-06T18:58:29Z)
• EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference [49.94169109038806]
  This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes.<n>Our results demonstrate at most 52.4% improvement in prefill throughput compared to existing parallel inference methods.
  arXiv  Detail & Related papers  (2024-10-16T05:17:49Z)
• Probabilistic electric load forecasting through Bayesian Mixture Density Networks [70.50488907591463]
  Probabilistic load forecasting (PLF) is a key component in the extended tool-chain required for efficient management of smart energy grids. We propose a novel PLF approach, framed on Bayesian Mixture Density Networks. To achieve reliable and computationally scalable estimators of the posterior distributions, both Mean Field variational inference and deep ensembles are integrated.
  arXiv  Detail & Related papers  (2020-12-23T16:21:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.