BeLLMan: Controlling LLM Congestion

• URL: http://arxiv.org/abs/2510.15330v1
• Date: Fri, 17 Oct 2025 05:36:42 GMT
• Title: BeLLMan: Controlling LLM Congestion
• Authors: Tella Rajashekhar Reddy, Atharva Deshmukh, Karan Tandon, Rohan Gandhi, Anjaly Parayil, Debopam Bhattacherjee,
• Abstract summary: Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load.<n>Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load.
• Score: 1.6728793271113227
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language model (LLM) applications are blindfolded to the infrastructure underneath and generate tokens autoregressively, indifferent to the system load, thus risking inferencing latency inflation and poor user experience. Our first-cut controller, named beLLMan, enables the LLM infrastructure to actively and progressively signal the first-party LLM application to adjust the output length in response to changing system load. On a real testbed with H100 GPUs, beLLMan helps keep inferencing latency under control (upto 8X lower end-to-end latency) and reduces energy consumption by 25% (while serving 19% more requests) during periods of congestion for a summarization workload.

Related papers

• lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models [7.524517279167586]
  Large Language Models (LLMs) are increasingly integrated into everyday applications.<n>Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs.<n>We propose lm-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference.
  arXiv  Detail & Related papers  (2025-10-07T17:05:30Z)
• VoltanaLLM: Feedback-Driven Frequency Control and State-Space Routing for Energy-Efficient LLM Serving [13.494819588196371]
  VoltanaLLM is a system for energy-efficient Large Language Model (LLM) serving.<n>It co-designs frequency scaling and request routing in emerging prefill/decode disaggregated architectures.<n>It achieves up to 36.3% energy savings while maintaining near-perfect SLO attainment rate.
  arXiv  Detail & Related papers  (2025-09-05T05:58:16Z)
• BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Serving [3.620158146761518]
  BucketServe is a bucket-based dynamic framework designed to optimize inference performance.<n>It can handle 1.93x more request load load attainment of 80% compared with UELLM and demonstrates 1.975x higher system load capacity compared to the UELLM.
  arXiv  Detail & Related papers  (2025-07-23T01:51:48Z)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models [54.1231638555233]
  Low-rank Sparse Adaptation (LoSA) is a novel method that seamlessly integrates low-rank adaptation into sparse LLM sparsity.<n>LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning.<n>LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden.
  arXiv  Detail & Related papers  (2025-02-20T18:37:32Z)
• Length Controlled Generation for Black-box LLMs [70.57649832433451]
  Large language models (LLMs) have demonstrated impressive instruction following capabilities, but struggle to accurately manage the length of generated text.<n>We propose a novel iterative sampling framework for text length control, integrating the Metropolis-Hastings algorithm with an importance sampling acceleration strategy.<n>Our framework achieves almost 100% success rates of length control on Llama3.1 for tasks such as length-controlled abstractive summarization.
  arXiv  Detail & Related papers  (2024-12-19T09:07:38Z)
• From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning [91.79567270986901]
  Large Language Models (LLMs) tend to prioritize adherence to user prompts over providing veracious responses.<n>Recent works propose to employ supervised fine-tuning (SFT) to mitigate the sycophancy issue.<n>We propose a novel supervised pinpoint tuning (SPT), where the region-of-interest modules are tuned for a given objective.
  arXiv  Detail & Related papers  (2024-09-03T07:01:37Z)
• Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decoding [61.45448947483328]
  We introduce Lossless Acceleration via Speculative Decoding for LLM-based Recommender Systems (LASER)<n>LASER features a Customized Retrieval Pool to enhance retrieval efficiency and Relaxed Verification to improve the acceptance rate of draft tokens.<n>LASER achieves a 3-5x speedup on public datasets and saves about 67% of computational resources during the online A/B test.
  arXiv  Detail & Related papers  (2024-08-11T02:31:13Z)
• Designing Efficient LLM Accelerators for Edge Devices [1.4128048241287314]
  Large Language Models (LLMs) can be deployed on resource-constrained edge devices to reduce reliance on network connections and provide more privacy. To address this issue, designing new and efficient edge accelerators for LLM inference is crucial. We propose SECDA-LLM, that utilizes the SECDA methodology to streamline the process of designing, integrating, and deploying efficient FPGA-based LLM accelerators.
  arXiv  Detail & Related papers  (2024-08-01T11:06:05Z)
• MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT [87.4910758026772]
  "Bigger the better" has been the predominant trend in recent Large Language Models (LLMs) development. This paper explores the "less is more" paradigm by addressing the challenge of designing accurate yet efficient Small Language Models (SLMs) for resource constrained devices.
  arXiv  Detail & Related papers  (2024-02-26T18:59:03Z)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs [67.38165028487242]
  We introduce Dynamic Sparse No Training (DSnoT), a training-free fine-tuning approach to fine-tune large language models (LLMs) Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs.
  arXiv  Detail & Related papers  (2023-10-13T07:38:52Z)
• Fast Distributed Inference Serving for Large Language Models [12.703624317418237]
  We present FastServe, a distributed inference serving system for large language models (LLMs) FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. We build a system prototype of FastServe and experimental results show that compared to the state-of-the-art solution vLLM, FastServe improves the throughput by up to 31.4x and 17.9x under the same average and tail latency requirements, respectively.
  arXiv  Detail & Related papers  (2023-05-10T06:17:50Z)
• Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networks [44.37047471448793]
  In this paper, we advocate the integration of edge computing paradigm and parallel split learning (PSL) We propose an innovative PSL framework, namely, efficient parallel split learning (EPSL) to accelerate model training. We show that the proposed EPSL framework significantly decreases the training latency needed to achieve a target accuracy.
  arXiv  Detail & Related papers  (2023-03-26T16:09:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.