Kalypso: Relational LLM Serving

• URL: http://arxiv.org/abs/2607.23815v1
• Date: Sun, 26 Jul 2026 19:29:31 GMT
• Title: Kalypso: Relational LLM Serving
• Abstract summary: Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan.<n>We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm.<n>Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads.
• Score: 6.264258565843104
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are increasingly used as semantic operators for filtering, extracting, ranking, joining, and transforming unstructured data. Existing semantic query processing systems invoke request-centric LLM serving systems that are unaware of the query plan, leaving substantial performance opportunities unused. This paper introduces relational LLM serving, an abstraction that makes LLM serving aware of semantic query structure while preserving query semantics and output accuracy. The key opportunity is pipelined execution across semantic operators: when intermediate tuples flow directly from one operator to the next, their KV-cache state can be reused instead of recomputed. We present Kalypso, a relational LLM serving system that exposes an API for semantic query plans and executes them using an adaptive, memory-aware scheduling algorithm. Kalypso addresses a new online scheduling problem in which pipelined operator execution is coupled with GPU memory pressure management to reuse KV-cache state in the serving engine before eviction. Its scheduler continuously adjusts memory allocations to balance upstream parallelism, downstream progress, and GPU utilization. Our evaluation shows that Kalypso improves query completion time over baselines using request-centric LLM serving, with speedups up to 4.57x across diverse workloads, demonstrating that query-aware LLM serving can substantially improve the efficiency of semantic query execution.

Related papers

• Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendations [53.53864188270018]
  Large language models (LLMs) have advanced recommendation systems (RSs)<n>We propose MARC to explicitly control the modularity of LLMs.<n>MARC achieved a 2.82% eCPM lift in an online A/B test within a large-scale commercial search advertising scenario.
  arXiv  Detail & Related papers  (2026-04-20T12:08:58Z)
• Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines [0.7551404527457958]
  We describe a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic onto a GPU cluster.<n>Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently.
  arXiv  Detail & Related papers  (2026-04-16T16:15:29Z)
• MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning [78.46301394559903]
  Large Language Models (LLMs) are increasingly used for long-duration tasks.<n>Current methods face a trade-off between cost and accuracy.<n>MemSifter is a novel framework that offloads the memory retrieval process to a small-scale proxy model.
  arXiv  Detail & Related papers  (2026-03-03T02:57:38Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• LLM-AutoDiff: Auto-Differentiate Any LLM Workflow [58.56731133392544]
  We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE)<n>LLMs-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine to generate feedback-akin to textual gradients.<n>It consistently outperforms existing textual gradient baselines in both accuracy and training cost.
  arXiv  Detail & Related papers  (2025-01-28T03:18:48Z)
• KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving [5.090882129969119]
  This paper proposes the first parameter-centric approach to handling throttling by selectively dropping replicated parameters to instantly free memory for requests.<n>We show that sys reduces the tail TTFT of requests under throttling by up to 72.2 times compared to the state-of-the-art systems including Llumnix, vLLM and InferCept.
  arXiv  Detail & Related papers  (2024-12-24T05:07:46Z)
• ALISE: Accelerating Large Language Model Serving with Speculative Scheduling [7.367068885621016]
  Large Language Models (LLMs) represent a revolutionary advancement in the contemporary landscape of artificial general intelligence (AGI) In this paper, we propose a new efficient LLM inference serving framework, named ALISE. We show that ALISE improves the throughput of inference serving by up to 1.8x and 2.1x under the same latency constraint on the Alpaca and ShareGPT datasets, respectively.
  arXiv  Detail & Related papers  (2024-10-31T00:58:11Z)
• Efficient LLM Scheduling by Learning to Rank [19.33941579312897]
  We show that it is possible to predict the relative ranks of output lengths in a batch of requests, using learning to rank. We develop a novel scheduler for LLM inference and serving that can approximate the shortest-job-first (SJF) schedule better than existing approaches.
  arXiv  Detail & Related papers  (2024-08-28T13:35:54Z)
• Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decoding [61.45448947483328]
  We introduce Lossless Acceleration via Speculative Decoding for LLM-based Recommender Systems (LASER)<n>LASER features a Customized Retrieval Pool to enhance retrieval efficiency and Relaxed Verification to improve the acceptance rate of draft tokens.<n>LASER achieves a 3-5x speedup on public datasets and saves about 67% of computational resources during the online A/B test.
  arXiv  Detail & Related papers  (2024-08-11T02:31:13Z)
• Optimizing LLM Queries in Relational Data Analytics Workloads [50.95919232839785]
  Batch data analytics is a growing application for Large Language Models (LLMs)<n>LLMs enable users to perform a wide range of natural language tasks, such as classification, entity extraction, and translation, over large datasets.<n>We propose novel techniques that can significantly reduce the cost of LLM calls for relational data analytics workloads.
  arXiv  Detail & Related papers  (2024-03-09T07:01:44Z)
• Fast Distributed Inference Serving for Large Language Models [12.703624317418237]
  We present FastServe, a distributed inference serving system for large language models (LLMs) FastServe exploits the autoregressive pattern of LLM inference to enable preemption at the granularity of each output token. We build a system prototype of FastServe and experimental results show that compared to the state-of-the-art solution vLLM, FastServe improves the throughput by up to 31.4x and 17.9x under the same average and tail latency requirements, respectively.
  arXiv  Detail & Related papers  (2023-05-10T06:17:50Z)
• Large Language Models are Strong Zero-Shot Retriever [89.16756291653371]
  We propose a simple method that applies a large language model (LLM) to large-scale retrieval in zero-shot scenarios. Our method, the Language language model as Retriever (LameR), is built upon no other neural models but an LLM.
  arXiv  Detail & Related papers  (2023-04-27T14:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.