Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges

• URL: http://arxiv.org/abs/2511.11624v1
• Date: Fri, 07 Nov 2025 02:58:15 GMT
• Title: Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges
• Abstract summary: This study evaluates the power efficiency of five representative SLMs on Raspberry Pi 5, Jetson Nano, TinyLlama, and Gemma 2.<n>Results show that Jetson Orin Nano with GPU acceleration achieves the highest energy-to-performance ratio.
• Score: 8.45935156679082
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cloud-based large language models (LLMs) and their variants have significantly influenced real-world applications. Deploying smaller models (i.e., small language models (SLMs)) on edge devices offers additional advantages, such as reduced latency and independence from network connectivity. However, edge devices' limited computing resources and constrained energy budgets challenge efficient deployment. This study evaluates the power efficiency of five representative SLMs - Llama 3.2, Phi-3 Mini, TinyLlama, and Gemma 2 on Raspberry Pi 5, Jetson Nano, and Jetson Orin Nano (CPU and GPU configurations). Results show that Jetson Orin Nano with GPU acceleration achieves the highest energy-to-performance ratio, significantly outperforming CPU-based setups. Llama 3.2 provides the best balance of accuracy and power efficiency, while TinyLlama is well-suited for low-power environments at the cost of reduced accuracy. In contrast, Phi-3 Mini consumes the most energy despite its high accuracy. In addition, GPU acceleration, memory bandwidth, and model architecture are key in optimizing inference energy efficiency. Our empirical analysis offers practical insights for AI, smart systems, and mobile ad-hoc platforms to leverage tradeoffs from accuracy, inference latency, and power efficiency in energy-constrained environments.

Related papers

• Evaluating the Energy Efficiency of NPU-Accelerated Machine Learning Inference on Embedded Microcontrollers [0.0]
  This paper evaluates the impact of Neural Processing Units (NPUs) on machine learning (ML) execution on microcontrollers (MCUs)<n>It shows substantial efficiency gains when inference is offloaded to the NPU.<n>For moderate to large networks, latency improvements ranged from 7x to over 125x, with per-inference net energy reductions up to 143x.
  arXiv  Detail & Related papers  (2025-09-22T08:52:54Z)
• Benchmarking Energy and Latency in TinyML: A Novel Method for Resource-Constrained AI [0.0]
  This work introduces an alternative benchmarking methodology that integrates energy and latency measurements.<n>To evaluate our setup, we tested the STM32N6 MCU, which includes a NPU for executing neural networks.<n>Our findings demonstrate that reducing the core voltage and clock frequency improve the efficiency of pre- and post-processing.
  arXiv  Detail & Related papers  (2025-05-21T15:12:14Z)
• Harnessing On-Device Large Language Model: Empirical Results and Implications for AI PC [8.837470787975308]
  Large Language Models (LLMs) on edge devices offer significant privacy benefits.<n>These on-device LLMs inherently face performance limitations due to reduced model capacity and necessary compression techniques.<n>We introduce a systematic methodology -- encompassing model capability, development efficiency, and system resources -- for evaluating on-device LLMs.
  arXiv  Detail & Related papers  (2025-05-21T02:23:01Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2 [5.213433310722838]
  Large language models (LLMs) deliver impressive performance but require large amounts of energy.<n>We present a MatMul-free LLM architecture adapted for Intel's neuromorphic processor, Loihi 2.<n>Our approach leverages Loihi 2's support for low-precision, event-driven computation and stateful processing.
  arXiv  Detail & Related papers  (2025-02-12T02:40:44Z)
• Benchmarking Deep Learning Models for Object Detection on Edge Computing Devices [0.0]
  We evaluate state-of-the-art object detection models, including YOLOv8 (Nano, Small, Medium), EfficientDet Lite (Lite0, Lite1, Lite2), and SSD (SSD MobileNet V1, SSDLite MobileDet) We deployed these models on popular edge devices like the Raspberry Pi 3, 4, and 5 with/without TPU accelerators, and Jetson Orin Nano, collecting key performance metrics such as energy consumption, inference time, and Mean Average Precision (mAP) Our findings highlight that lower mAP models such as SSD MobileNet V1 are more energy-efficient and faster in
  arXiv  Detail & Related papers  (2024-09-25T10:56:49Z)
• Latency-aware Unified Dynamic Networks for Efficient Image Recognition [72.8951331472913]
  LAUDNet is a framework to bridge the theoretical and practical efficiency gap in dynamic networks. It integrates three primary dynamic paradigms-spatially adaptive computation, dynamic layer skipping, and dynamic channel skipping. It can notably reduce the latency of models like ResNet by over 50% on platforms such as V100,3090, and TX2 GPUs.
  arXiv  Detail & Related papers  (2023-08-30T10:57:41Z)
• Data-Model-Circuit Tri-Design for Ultra-Light Video Intelligence on Edge Devices [90.30316433184414]
  We propose a data-model-hardware tri-design framework for high- throughput, low-cost, and high-accuracy MOT on HD video stream. Compared to the state-of-the-art MOT baseline, our tri-design approach can achieve 12.5x latency reduction, 20.9x effective frame rate improvement, 5.83x lower power, and 9.78x better energy efficiency, without much accuracy drop.
  arXiv  Detail & Related papers  (2022-10-16T16:21:40Z)
• Faster Attention Is What You Need: A Fast Self-Attention Neural Network Backbone Architecture for the Edge via Double-Condensing Attention Condensers [71.40595908386477]
  We introduce a new faster attention condenser design called double-condensing attention condensers. The resulting backbone (which we name AttendNeXt) achieves significantly higher inference throughput on an embedded ARM processor. These promising results demonstrate that exploring different efficient architecture designs and self-attention mechanisms can lead to interesting new building blocks for TinyML applications.
  arXiv  Detail & Related papers  (2022-08-15T02:47:33Z)
• FPGA-optimized Hardware acceleration for Spiking Neural Networks [69.49429223251178]
  This work presents the development of a hardware accelerator for an SNN, with off-line training, applied to an image recognition task. The design targets a Xilinx Artix-7 FPGA, using in total around the 40% of the available hardware resources. It reduces the classification time by three orders of magnitude, with a small 4.5% impact on the accuracy, if compared to its software, full precision counterpart.
  arXiv  Detail & Related papers  (2022-01-18T13:59:22Z)
• FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation [81.76975488010213]
  Dense optical flow estimation plays a key role in many robotic vision tasks. Current networks often occupy large number of parameters and require heavy computation costs. Our proposed FastFlowNet works in the well-known coarse-to-fine manner with following innovations.
  arXiv  Detail & Related papers  (2021-03-08T03:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.