The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving

• URL: http://arxiv.org/abs/2405.11299v2
• Date: Mon, 27 May 2024 01:09:07 GMT
• Title: The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving
• Authors: Pai Zeng, Zhenyu Ning, Jieru Zhao, Weihao Cui, Mengwei Xu, Liwei Guo, Xusheng Chen, Yizhou Shan,
• Abstract summary: We survey the large language model (LLM) serving area to understand the dynamics between cost-efficiency and accuracy. Our findings reveal that works in this space optimize along three distinct but conflicting goals: improving serving context length (C), improving serving accuracy (A), and improving serving performance (P)
• Score: 5.199465146247884
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer contextual understanding when deploying models at a massive scale. Our findings reveal that works in this space optimize along three distinct but conflicting goals: improving serving context length (C), improving serving accuracy (A), and improving serving performance (P). Drawing inspiration from the CAP theorem in databases, we propose a CAP principle for LLM serving, which suggests that any optimization can improve at most two of these three goals simultaneously. Our survey categorizes existing works within this framework. We find the definition and continuity of user-perceived measurement metrics are crucial in determining whether a goal has been met, akin to prior CAP databases in the wild. We recognize the CAP principle for LLM serving as a guiding principle, rather than a formal theorem, to inform designers of the inherent and dynamic trade-offs in serving models. As serving accuracy and performance have been extensively studied, this survey focuses on works that extend serving context length and address the resulting challenges.

Related papers

• General-Reasoner: Advancing LLM Reasoning Across All Domains [64.70599911897595]
  Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs)<n>We propose General-Reasoner, a novel training paradigm designed to enhance LLM reasoning capabilities across diverse domains.<n>We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc.
  arXiv  Detail & Related papers  (2025-05-20T17:41:33Z)
• Evaluating the Goal-Directedness of Large Language Models [17.08087240111954]
  We evaluate goal-directedness on tasks that require information gathering, cognitive effort, and plan execution. Our evaluations of LLMs from Google DeepMind, OpenAI, and Anthropic show that goal-directedness is relatively consistent across tasks.
  arXiv  Detail & Related papers  (2025-04-16T08:07:08Z)
• Goal2Story: A Multi-Agent Fleet based on Privately Enabled sLLMs for Impacting Mapping on Requirements Elicitation [6.547589336272875]
  Goal2Story is a multi-agent fleet that adopts the Impact Mapping (IM) framework while merely using cost-effective sLLMs for goal-driven RE. StorySeek dataset contains over 1,000 user stories (USs) with corresponding goals and project context information. For evaluation, we proposed two metrics: Factuality Hit Rate (FHR) to measure consistency between the generated USs with the dataset and Quality And Consistency Evaluation (QuACE) to evaluate the quality of the generated USs.
  arXiv  Detail & Related papers  (2025-03-17T15:31:20Z)
• Large Reasoning Models in Agent Scenarios: Exploring the Necessity of Reasoning Capabilities [74.35956310688164]
  We propose the LaRMA framework, encompassing nine tasks across Tool Usage, Plan Design, and Problem Solving. Our findings address four research questions: LRMs surpass LLMs in reasoning-intensive tasks like Plan Design, leveraging iterative reflection for superior outcomes. LRMs' enhanced reasoning incurs higher computational costs, prolonged processing, and behavioral challenges, including overthinking and fact-ignoring tendencies.
  arXiv  Detail & Related papers  (2025-03-14T04:34:31Z)
• Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Training [69.13064064991552]
  Hephaestus-Forge is a large-scale pre-training corpus designed to enhance the capabilities of LLM agents in API function calling, intrinsic reasoning and planning. Hephaestus-Forge comprises 103B agent-specific data encompassing 76,537 APIs, including both tool documentation to introduce knowledge of API functions and function calling trajectories. By continual pre-training on Hephaestus-Forge, Hephaestus outperforms small- to medium-scale open-source LLMs and rivals commercial LLMs on three agent benchmarks.
  arXiv  Detail & Related papers  (2025-02-10T15:54:34Z)
• Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions [59.5243730853157]
  Federated learning (FL) provides a privacy-preserving solution for fine-tuning pre-trained large language models (LLMs) using distributed private datasets. This article conducts a comparative analysis of three advanced federated LLM (FedLLM) frameworks that integrate knowledge distillation (KD) and split learning (SL) to mitigate these issues.
  arXiv  Detail & Related papers  (2025-01-08T11:37:06Z)
• Adaptive Pruning for Large Language Models with Structural Importance Awareness [66.2690963378878]
  Large language models (LLMs) have significantly improved language understanding and generation capabilities. LLMs are difficult to deploy on resource-constrained edge devices due to their high computational and storage resource demands. We propose structurally-aware adaptive pruning (SAAP) to significantly reduce the computational and memory costs while maintaining model performance.
  arXiv  Detail & Related papers  (2024-12-19T18:08:04Z)
• Are Your LLMs Capable of Stable Reasoning? [38.03049704515947]
  We introduce G-Pass@$k$, a novel evaluation metric that continuously assesses model performance across multiple sampling attempts.<n>We employ G-Pass@$k$ in conjunction with state-of-the-art large language models to provide comprehensive insights into their potential capabilities and operational consistency.
  arXiv  Detail & Related papers  (2024-12-17T18:12:47Z)
• Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate [118.37653302885607]
  We present the Modality Integration Rate (MIR), an effective, robust, and generalized metric to indicate the multi-modal pre-training quality of Large Vision Language Models (LVLMs) MIR is indicative about training data selection, training strategy schedule, and model architecture design to get better pre-training results.
  arXiv  Detail & Related papers  (2024-10-09T17:59:04Z)
• Rational Metareasoning for Large Language Models [5.5539136805232205]
  Being prompted to engage in reasoning has emerged as a core technique for using large language models (LLMs) This work introduces a novel approach based on computational models of metareasoning used in cognitive science. We develop a reward function that incorporates the Value of Computation by penalizing unnecessary reasoning.
  arXiv  Detail & Related papers  (2024-10-07T23:48:52Z)
• VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment [66.80143024475635]
  We propose VinePPO, a straightforward approach to compute unbiased Monte Carlo-based estimates. We show that VinePPO consistently outperforms PPO and other RL-free baselines across MATH and GSM8K datasets.
  arXiv  Detail & Related papers  (2024-10-02T15:49:30Z)
• Benchmarking General-Purpose In-Context Learning [19.40952728849431]
  In-context learning (ICL) empowers generative models to address new tasks effectively and efficiently on the fly. In this paper, we study extending ICL to address a broader range of tasks with an extended learning horizon and higher improvement potential. We introduce two benchmarks specifically crafted to train and evaluate GPICL functionalities.
  arXiv  Detail & Related papers  (2024-05-27T14:50:42Z)
• LLM In-Context Recall is Prompt Dependent [0.0]
  A model's ability to do this significantly influences its practical efficacy and dependability in real-world applications. This study demonstrates that an LLM's recall capability is not only contingent upon the prompt's content but also may be compromised by biases in its training data.
  arXiv  Detail & Related papers  (2024-04-13T01:13:59Z)
• Dynamic Evaluation of Large Language Models by Meta Probing Agents [44.20074234421295]
  We propose meta probing agents (MPA) to evaluate large language models (LLMs) MPA is the key component of DyVal 2, which naturally extends the previous DyValcitepzhu2023dyval. MPA designs the probing and judging agents to automatically transform an original evaluation problem into a new one following psychometric theory.
  arXiv  Detail & Related papers  (2024-02-21T06:46:34Z)
• A Principled Framework for Knowledge-enhanced Large Language Model [58.1536118111993]
  Large Language Models (LLMs) are versatile, yet they often falter in tasks requiring deep and reliable reasoning. This paper introduces a rigorously designed framework for creating LLMs that effectively anchor knowledge and employ a closed-loop reasoning process.
  arXiv  Detail & Related papers  (2023-11-18T18:10:02Z)
• Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models [61.28463542324576]
  Vision-language models (VLMs) have recently demonstrated strong efficacy as visual assistants that can generate human-like outputs. We evaluate existing state-of-the-art VLMs and find that even the best-performing model is unable to demonstrate strong visual reasoning capabilities and consistency. We propose a two-stage training framework aimed at improving both the reasoning performance and consistency of VLMs.
  arXiv  Detail & Related papers  (2023-09-08T17:49:44Z)
• Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty [52.72790059506241]
  Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
  arXiv  Detail & Related papers  (2023-09-07T01:35:24Z)
• Exploring Self-supervised Logic-enhanced Training for Large Language Models [59.227222647741094]
  In this paper, we make the first attempt to investigate the feasibility of incorporating logical knowledge through self-supervised post-training. We devise an auto-regressive objective variant of MERIt and integrate it with two LLM series, i.e., FLAN-T5 and LLaMA, with parameter size ranging from 3 billion to 13 billion. The results on two challenging logical reasoning benchmarks demonstrate the effectiveness of LogicLLM.
  arXiv  Detail & Related papers  (2023-05-23T06:13:10Z)
• Post Hoc Explanations of Language Models Can Improve Language Models [43.2109029463221]
  We present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY) We leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. Our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks.
  arXiv  Detail & Related papers  (2023-05-19T04:46:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.