Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management

• URL: http://arxiv.org/abs/2505.03756v1
• Date: Sat, 19 Apr 2025 13:17:34 GMT
• Title: Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
• Abstract summary: Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications.<n>Existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT)<n>FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper.
• Score: 13.327937177980969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.

Related papers

• ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cache [8.323914332215598]
  Low-Rank Adaptation (LoRA) introduces a critical memory footprint bottleneck during serving.<n>By exploiting the structural properties of LoRA, ForkKV physically decouples the KV cache into a massive shared component.<n>We show that ForkKV achieves up to 3.0x the throughput of state-of-the-art multi-LoRA serving systems.
  arXiv  Detail & Related papers  (2026-04-07T18:52:25Z)
• LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agents [9.162948089580143]
  We propose LRAgent, a KV cache sharing framework for multi-LoRA agents.<n> LRAgent decomposes the cache into a shared base component from the pretrained weights and an adapter-dependent component from LoRA weights.<n> LRAgent achieves throughput and time-to-first-token latency close to fully shared caching.
  arXiv  Detail & Related papers  (2026-02-01T06:36:46Z)
• LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference [27.24239725255976]
  We present LMCache, the first and so far the most efficient open-source KV caching solution.<n>It extracts and stores KV caches generated by modern LLM engines and shares the KV caches across engines and queries.<n>We show that combining LMCache with vLLM achieves up to 15x improvement in throughput across diverse workloads.
  arXiv  Detail & Related papers  (2025-10-08T00:15:04Z)
• Faster Than SVD, Smarter Than SGD: The OPLoRA Alternating Update [50.36542772932594]
  Low-Rank Adaptation (LoRA) fine-tunes large models by learning low-rank updates on top of frozen weights.<n>There is still a gap between full training with low-rank projections (SVDLoRA) and LoRA fine-tuning, indicating that LoRA steps can be further improved.
  arXiv  Detail & Related papers  (2025-09-24T10:32:50Z)
• LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Models [52.56008278458534]
  LaCache is a training-free method for efficient and accurate generative inference of Large Language Models.<n>LaCache enables LLMs to address both of the critical challenges in long-range modeling: robust long-range capabilities and continuous generation without running out-of-memory.
  arXiv  Detail & Related papers  (2025-07-14T19:09:57Z)
• Shared Disk KV Cache Management for Efficient Multi-Instance Inference in RAG-Powered LLMs [5.02504911036896]
  Recent large language models (LLMs) face increasing inference latency as input context length and model size grow.<n>This paper proposes a method to reduce TTFT by leveraging a disk-based key-value (KV) cache to lessen the computational burden during the prefill stage.<n>We also introduce a disk-based shared KV cache management system, called Shared RAG-DCache, for multi-instance LLM RAG service environments.
  arXiv  Detail & Related papers  (2025-04-16T04:59:18Z)
• A Generative Caching System for Large Language Models [1.2132389187658934]
  Caching has the potential to be of significant benefit for accessing large language models (LLMs)<n>This paper presents a new caching system for improving user experiences with LLMs.<n>A key feature we provide is generative caching, wherein multiple cached responses can be synthesized to provide answers to queries which have never been seen before.
  arXiv  Detail & Related papers  (2025-03-22T01:17:56Z)
• InstCache: A Predictive Cache for LLM Serving [6.076957323090607]
  Caching techniques offer opportunities to optimize the performance of Large Language Models inference engines.<n>High variability in the content and length of instructions make it rare for identical instructions to recur within a short time window.<n>We propose InstCache, a predictive caching mechanism for LLM serving systems.
  arXiv  Detail & Related papers  (2024-11-21T03:52:41Z)
• LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization [78.93425154518705]
  Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements.<n>This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization.
  arXiv  Detail & Related papers  (2024-10-27T22:57:12Z)
• ThinK: Thinner Key Cache by Query-Driven Pruning [63.13363917871414]
  Large Language Models (LLMs) have revolutionized the field of natural language processing, achieving unprecedented performance across a variety of applications.<n>This paper focuses on the long-context scenario, addressing the inefficiencies in KV cache memory consumption during inference.<n>We propose ThinK, a novel query-dependent KV cache pruning method designed to minimize attention weight loss while selectively pruning the least significant channels.
  arXiv  Detail & Related papers  (2024-07-30T17:59:08Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters [59.490751234925206]
  Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is often employed to adapt a base model to a multitude of tasks. We present S-LoRA, a system designed for the scalable serving of many LoRA adapters.
  arXiv  Detail & Related papers  (2023-11-06T17:26:17Z)
• CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices [78.16679232748196]
  We introduce a Compression-Aware LoRA (CA-LoRA) framework to transfer Large Language Models (LLMs) to other tasks. Experiment results demonstrate that CA-LoRA outperforms the vanilla LoRA methods applied to a compressed LLM. The source code of CA-LoRA is available at https://github.com/thunlp/CA-LoRA.
  arXiv  Detail & Related papers  (2023-07-15T04:37:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.