A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware

• URL: http://arxiv.org/abs/2609.08307v1
• Date: Tue, 08 Sep 2026 06:28:32 GMT
• Title: A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
• Abstract summary: Large language models (LLMs) are increasingly used as backends for intelligent web services.<n>This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes.
• Score: 0.14913052010438635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services.

Related papers

• Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference [23.424767173051404]
  Vision-Language-Action (VLA) models have achieved impressive performance on diverse embodied tasks.<n> deploying VLA models on low-power onboard devices, such as the Jetson Orin, remains challenging due to their high computational complexity.<n>Asynchronous inference can partially mask this latency by parallelizing action execution and subsequent inference, but it introduces two critical issues: perception-execution misalignment and long reaction time.<n>We propose Jetson-PI, a method for efficient VLA deployment on onboard devices via Foresight-Aligned Asynchronous Correction.
  arXiv  Detail & Related papers  (2026-07-14T11:38:36Z)
• DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners? [57.585275546688116]
  Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents.<n>We argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world.<n>We introduce DIRECT, a routing framework that uses multimodal scene context to allocate compute per prompt.
  arXiv  Detail & Related papers  (2026-06-10T17:58:49Z)
• Edge-Based QoS-Aware Adaptive Task Placement: A Closed-Loop Control in Multi-Robot Systems [5.480084037947013]
  We present a pilot edge-centric MRS testbed using Raspberry Pi nodes to evaluate a camera-to-manipulator pipeline.<n>Results show that ATP-aware controller, by switching task placement based on measured latency and utilization thresholds, consistently lowers deadline violations and tail latency.
  arXiv  Detail & Related papers  (2026-05-30T05:54:44Z)
• Profiling-Driven Adaptive Distributed Transformer Inference on Embedded Edge Deployment [60.442064966340524]
  We present a prototype study on NVIDIA Jetson Orin Nano devices connected over WiFi.<n>Key finding is that the dominant bottleneck is not just network bandwidth but also the CPU- GPU staging during communication.<n>Experiments show that this strategy reduces latency by 65%-77% and energy consumption by 34%-52% relative to full-tensor exchange.
  arXiv  Detail & Related papers  (2026-05-25T10:39:28Z)
• Serverless GPU Architecture for Enterprise HR Analytics: A Production-Scale BDaaS Implementation [6.240627892585199]
  We present a production-oriented Big Data as a Service (BD) blueprint that integrates a singlenode serverless GPU runtime with TabNet.<n>We conduct benchmarks on the HR, Adult, and BLS datasets, comparing our approach against Spark and CPU baselines.<n>Our results show that GPU pipelines achieve up to 4.5x higher throughput, 98x lower latency, and 90% lower cost per 1K inferences compared to Spark baselines.
  arXiv  Detail & Related papers  (2025-10-22T15:37:42Z)
• lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models [7.524517279167586]
  Large Language Models (LLMs) are increasingly integrated into everyday applications.<n>Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs.<n>We propose lm-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference.
  arXiv  Detail & Related papers  (2025-10-07T17:05:30Z)
• The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks [56.37880529653111]
  The demand for large computation model (LAIM) services is driving a paradigm shift from traditional cloud-based inference to edge-based inference for low-latency, privacy-preserving applications.<n>In this paper, we investigate the LAIM-inference scheme, where a pre-trained LAIM is pruned and partitioned into on-device and on-server sub-models for deployment.
  arXiv  Detail & Related papers  (2025-05-14T08:18:55Z)
• MAPLE-X: Latency Prediction with Explicit Microprocessor Prior Knowledge [87.41163540910854]
  Deep neural network (DNN) latency characterization is a time-consuming process. We propose MAPLE-X which extends MAPLE by incorporating explicit prior knowledge of hardware devices and DNN architecture latency.
  arXiv  Detail & Related papers  (2022-05-25T11:08:20Z)
• MAPLE-Edge: A Runtime Latency Predictor for Edge Devices [80.01591186546793]
  We propose MAPLE-Edge, an edge device-oriented extension of MAPLE, the state-of-the-art latency predictor for general purpose hardware. Compared to MAPLE, MAPLE-Edge can describe the runtime and target device platform using a much smaller set of CPU performance counters. We also demonstrate that unlike MAPLE which performs best when trained on a pool of devices sharing a common runtime, MAPLE-Edge can effectively generalize across runtimes.
  arXiv  Detail & Related papers  (2022-04-27T14:00:48Z)
• Towards Streaming Perception [70.68520310095155]
  We present an approach that coherently integrates latency and accuracy into a single metric for real-time online perception. The key insight behind this metric is to jointly evaluate the output of the entire perception stack at every time instant. We focus on the illustrative tasks of object detection and instance segmentation in urban video streams, and contribute a novel dataset with high-quality and temporally-dense annotations.
  arXiv  Detail & Related papers  (2020-05-21T01:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.