SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud

• URL: http://arxiv.org/abs/2602.23722v1
• Date: Fri, 27 Feb 2026 06:43:47 GMT
• Title: SLA-Aware Distributed LLM Inference Across Device-RAN-Cloud
• Abstract summary: Embodied AI requires sub-second inference near the Radio Access Network (RAN)<n>We report measurements from a 5G Standalone (SA) AI-RAN testbed using a fixed baseline policy for repeatability.
• Score: 44.19683744405876
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embodied AI requires sub-second inference near the Radio Access Network (RAN), but deployments span heterogeneous tiers (on-device, RAN-edge, cloud) and must not disrupt real-time baseband processing. We report measurements from a 5G Standalone (SA) AI-RAN testbed using a fixed baseline policy for repeatability. The setup includes an on-device tier, a three-node RAN-edge cluster co-hosting a containerized 5G RAN, and a cloud tier. We find that on-device execution remains multi-second and fails to meet sub-second budgets. At the RAN edge, SLA feasibility is primarily determined by model variant choice: quantized models concentrate below 0.5\,s, while unquantized and some larger quantized models incur deadline misses due to stalls and queuing. In the cloud tier, meeting a 0.5\,s deadline is challenging on the measured WAN path (up to 32.9\% of requests complete within 0.5\,s), but all evaluated variants meet a 1.0\,s deadline (100\% within 1.0\,s). Under saturated downlink traffic and up to $N{=}20$ concurrent inference clients, Multi-Instance GPU (MIG) isolation preserves baseband timing-health proxies, supporting safe co-location under fixed partitioning.

Related papers

• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving [12.697860184583483]
  Co-batching improves throughput but obscures per-tenant usage and limits control.<n>We present LLMVisor, a roofline-guided latency model that captures the memory-bound and compute-bound phases.<n>Compared to a token-count baseline, LLMVisor attains near-perfect R-squared and reduces relative error by up to 2.5x.
  arXiv  Detail & Related papers  (2026-08-09T00:28:05Z)
• AUGUSTE: Online-Learning dApp for Predictive URLLC Scheduling [10.973028847423807]
  Real 5G networks still show median Uplink (UL) round-trip times in the 50-70 ms range.<n>We propose AUGUSTE, a learning-based Medium Access Control (MAC) scheduling framework that embeds online Machine Learning (ML) models in the UL scheduler.<n>We evaluate AUGUSTE on a real 5G testbed running OpenAirInterface across three URLLC traffic patterns.
  arXiv  Detail & Related papers  (2026-06-02T13:50:22Z)
• Benchmarking Inductive Biases for Multivariate Time-Series Anomaly Detection with a Robust Multi-View Channel-Graph Detector [8.29991973809832]
  Ten family-representative detectors are evaluated on five datasets.<n>All methods share the same windowing, scoring, hardware, and metric protocols.<n>We introduce ours, an adaptive detector family combining a NOTEARS-constrained directed channel-graph view with optional patch-attention and temporal-association views.
  arXiv  Detail & Related papers  (2026-05-27T07:55:44Z)
• HALO: Semantic-Aware Distributed LLM Inference in Lossy Edge Network [50.33808558714122]
  Large language models' (LLMs) inference at the edge can facilitate prompt service responsiveness while protecting user privacy.<n>We propose HALO, a novel framework that can boost the distributed LLM inference in lossy edge network.<n> Experimental results from a Raspberry Pi cluster demonstrate that HALO achieves a 3.41x end-to-end speedup for LLaMA-series LLMs under unreliable network conditions.
  arXiv  Detail & Related papers  (2026-01-16T07:37:23Z)
• One-Shot Federated Ridge Regression: Exact Recovery via Sufficient Statistic Aggregation [0.7106986689736825]
  Federated ridge regression is a distributed equilibrium problem where each client computes local sufficient statistics and transmits them once.<n>We establish differential privacy guarantees where noise is injected once per client, eliminating the composition penalty that degrades privacy in multi-round protocols.<n>Experiments on synthetic heterogeneous regression demonstrate that one-shot fusion matches FedAvg accuracy while requiring up to $38times$ less communication.
  arXiv  Detail & Related papers  (2026-01-13T04:47:22Z)
• From Static to Dynamic: A Streaming RAG Approach to Real-time Knowledge Base [0.0]
  Streaming RAG is a unified pipeline that combines cosine screening, mini-batch clustering, and a heavy-hitter filter to maintain a compact prototype set.<n>Experiments on eight real-time streams show statistically significant gains in Recall@10 (up to 3 points, p 0.01), end-to-end latency below 15 ms, and throughput above 900 documents per second under a 150 MB budget.
  arXiv  Detail & Related papers  (2025-07-31T14:03:19Z)
• Distributed Training under Packet Loss [8.613477072763404]
  Leveraging unreliable connections will reduce latency but may sacrifice model accuracy and convergence once packets are dropped.<n>We introduce a principled, end-to-end solution that preserves accuracy and convergence guarantees under genuine packet loss.<n>This work bridges the gap between communication-efficient protocols and the accuracy and guarantees demanded by modern large-model training.
  arXiv  Detail & Related papers  (2025-07-02T11:07:20Z)
• Generative AI on the Edge: Architecture and Performance Evaluation [0.3999851878220877]
  6G's AI native vision of embedding advance intelligence in the network requires a systematic evaluation of Generative AI (GenAI) models on edge devices.<n>This research investigates computationally demanding Large Language Models (LLMs) inference on a single commodity Raspberry Pi serving as an edge testbed for ORAN.
  arXiv  Detail & Related papers  (2024-11-18T16:09:01Z)
• Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization [90.08459757321405]
  Federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead.<n>We propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates local model parameters and moment estimates.<n>By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error.
  arXiv  Detail & Related papers  (2024-05-28T07:56:49Z)
• DASA: Delay-Adaptive Multi-Agent Stochastic Approximation [64.32538247395627]
  We consider a setting in which $N$ agents aim to speedup a common Approximation problem by acting in parallel and communicating with a central server. To mitigate the effect of delays and stragglers, we propose textttDASA, a Delay-Adaptive algorithm for multi-agent Approximation.
  arXiv  Detail & Related papers  (2024-03-25T22:49:56Z)
• Peer-to-Peer Deep Learning for Beyond-5G IoT [10.425510512462628]
  P2PL is a multi-device peer-to-peer deep learning algorithm that does not require coordination from edge servers or the cloud. This makes P2PL well-suited for the sheer scale of beyond-5G computing environments like smart cities.
  arXiv  Detail & Related papers  (2023-10-29T01:18:45Z)
• Timely Asynchronous Hierarchical Federated Learning: Age of Convergence [59.96266198512243]
  We consider an asynchronous hierarchical federated learning setting with a client-edge-cloud framework. The clients exchange the trained parameters with their corresponding edge servers, which update the locally aggregated model. The goal of each client is to converge to the global model, while maintaining timeliness of the clients.
  arXiv  Detail & Related papers  (2023-06-21T17:39:16Z)
• Convergence of Update Aware Device Scheduling for Federated Learning at the Wireless Edge [84.55126371346452]
  We study federated learning at the wireless edge, where power-limited devices with local datasets collaboratively train a joint model with the help of a remote parameter server (PS) We design novel scheduling and resource allocation policies that decide on the subset of the devices to transmit at each round. The results of numerical experiments show that the proposed scheduling policy, based on both the channel conditions and the significance of the local model updates, provides a better long-term performance than scheduling policies based only on either of the two metrics individually.
  arXiv  Detail & Related papers  (2020-01-28T15:15:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.