ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs

• URL: http://arxiv.org/abs/2505.14468v1
• Date: Tue, 20 May 2025 15:04:17 GMT
• Title: ServerlessLoRA: Minimizing Latency and Cost in Serverless Inference for LoRA-Based LLMs
• Authors: Yifan Sui, Hao Wang, Hanfei Yu, Yitao Hu, Jianxun Li, Hao Wang,
• Abstract summary: Current serverless can effectively serve general Large Language Model (LLM) but fail with Low-Rank Adaptation (LoRA) inference.<n>These inefficiencies lead to massive GPU wastage, increased Time-To-First-Token (TTFT), and high monetary costs.<n>We propose ServerlessLoRA, a novel serverless inference system designed for faster and cheaper LoRA LLM serving.
• Score: 6.907528479144716
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Serverless computing has grown rapidly for serving Large Language Model (LLM) inference due to its pay-as-you-go pricing, fine-grained GPU usage, and rapid scaling. However, our analysis reveals that current serverless can effectively serve general LLM but fail with Low-Rank Adaptation (LoRA) inference due to three key limitations: 1) massive parameter redundancy among functions where 99% of weights are unnecessarily duplicated, 2) costly artifact loading latency beyond LLM loading, and 3) magnified resource contention when serving multiple LoRA LLMs. These inefficiencies lead to massive GPU wastage, increased Time-To-First-Token (TTFT), and high monetary costs. We propose ServerlessLoRA, a novel serverless inference system designed for faster and cheaper LoRA LLM serving. ServerlessLoRA enables secure backbone LLM sharing across isolated LoRA functions to reduce redundancy. We design a pre-loading method that pre-loads comprehensive LoRA artifacts to minimize cold-start latency. Furthermore, ServerlessLoRA employs contention aware batching and offloading to mitigate GPU resource conflicts during bursty workloads. Experiment on industrial workloads demonstrates that ServerlessLoRA reduces TTFT by up to 86% and cuts monetary costs by up to 89% compared to state-of-the-art LLM inference solutions.

Related papers

• Dynamic Low-Rank Sparse Adaptation for Large Language Models [54.1231638555233]
  Low-rank Sparse Adaptation (LoSA) is a novel method that seamlessly integrates low-rank adaptation into sparse LLM sparsity.<n>LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning.<n>LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden.
  arXiv  Detail & Related papers  (2025-02-20T18:37:32Z)
• Empower Vision Applications with LoRA LMM [32.37720746437661]
  Low-rank adaptation (LoRA) offers a promising method to integrate external knowledge into large language models (LMMs)<n>Existing LoRA model serving is excessively computationally expensive and causes extremely high latency.<n>We present an end-to-end solution that empowers diverse vision tasks and enriches vision applications with LoRA LMMs.
  arXiv  Detail & Related papers  (2024-11-01T13:43:33Z)
• Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning [57.36978335727009]
  Low-Rank Adaptation (LoRA) offers an efficient way to fine-tune large language models (LLMs) In this paper, we propose a framework that adaptively retrieves and composes multiple LoRAs based on input prompts.
  arXiv  Detail & Related papers  (2024-06-24T05:24:41Z)
• LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report [3.304521604464247]
  Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted methods for. Efficient Fine-Tuning (PEFT) of Large Language Models (LLMs) We aim to assess the viability of training and serving LLMs fine-tuned with LoRA in real-world applications.
  arXiv  Detail & Related papers  (2024-04-29T04:01:45Z)
• mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs [5.735411578779657]
  Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is commonly used to adapt a base LLM to multiple downstream tasks. LoRA platforms enable developers to fine-tune multiple models and develop various domain-specific applications simultaneously. Existing model parallelism schemes suffer from high communication overhead and inefficient GPU utilization when training multiple LoRA tasks.
  arXiv  Detail & Related papers  (2023-12-05T05:38:38Z)
• SpotServe: Serving Generative Large Language Models on Preemptible Instances [64.18638174004151]
  SpotServe is the first distributed large language models serving system on preemptible instances. We show that SpotServe can reduce the P99 tail latency by 2.4 - 9.1x compared with the best existing LLM serving systems. We also show that SpotServe can leverage the price advantage of preemptive instances, saving 54% monetary cost compared with only using on-demand instances.
  arXiv  Detail & Related papers  (2023-11-27T06:31:17Z)
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters [59.490751234925206]
  Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks. We present S-LoRA, a system designed for the scalable serving of many LoRA adapters.
  arXiv  Detail & Related papers  (2023-11-06T17:26:17Z)
• CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices [78.16679232748196]
  We introduce a Compression-Aware LoRA (CA-LoRA) framework to transfer Large Language Models (LLMs) to other tasks. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.
  arXiv  Detail & Related papers  (2023-07-15T04:37:11Z)
• LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning [56.88751562302793]
  Low-rank adaption (LoRA) has emerged to fine-tune large language models (LLMs) LoRAPrune is a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner. LoRAPrune achieves a reduction in perplexity by 4.81 on WikiText2 and 3.46 on PTB, while also decreasing memory usage by 52.6%.
  arXiv  Detail & Related papers  (2023-05-28T15:15:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.