SGLang: Efficient Execution of Structured Language Model Programs

• URL: http://arxiv.org/abs/2312.07104v2
• Date: Thu, 6 Jun 2024 00:10:06 GMT
• Title: SGLang: Efficient Execution of Structured Language Model Programs
• Abstract summary: We introduce SGLang, a system for efficient execution of complex language model programs. The runtime execution with novel optimizations like Radix for KVLang and compressed finite state machines for faster structured output decoding. Experiments that SG cache up to 6.4x higher throughput compared to state-of-the-art inference on various large language multi-modal models.
• Score: 38.7354780030876
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show that SGLang achieves up to 6.4x higher throughput compared to state-of-the-art inference systems on various large language and multi-modal models on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The code is publicly available at https://github.com/sgl-project/sglang

Related papers

• WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding [58.1177179119881]
  We introduce wgrammar, a lightweight decoding engine that integrates domain-aware simplification, constraint decomposition, and mask caching.<n> wgrammar achieves up to 250x speedup over existing systems.
  arXiv  Detail & Related papers  (2025-07-22T17:13:47Z)
• Type-Constrained Code Generation with Language Models [51.03439021895432]
  We introduce a type-constrained decoding approach that leverages type systems to guide code generation.<n>For this purpose, we develop novel prefix automata and a search over inhabitable types, forming a sound approach to enforce well-typedness on LLM-generated code.<n>Our approach reduces compilation errors by more than half and significantly increases functional correctness in code synthesis, translation, and repair tasks.
  arXiv  Detail & Related papers  (2025-04-12T15:03:00Z)
• XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models [3.9417976759908573]
  Context-free grammar is a flexible approach to enable structured generation via constrained decoding. XGrammar is a flexible and efficient structure generation engine for large language models. XGrammar can achieve up to 100x speedup over existing solutions.
  arXiv  Detail & Related papers  (2024-11-22T18:01:37Z)
• CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution [50.7413285637879]
  The CRUXEVAL-X code reasoning benchmark contains 19 programming languages. It comprises at least 600 subjects for each language, along with 19K content-consistent tests in total. Even a model trained solely on Python can achieve at most 34.4% Pass@1 in other languages.
  arXiv  Detail & Related papers  (2024-08-23T11:43:00Z)
• Dynamic Code Orchestration: Harnessing the Power of Large Language Models for Adaptive Script Execution [0.5735035463793009]
  Research examines dynamic code execution of written language directives within the context of a running application. The research clearly shows how written language directives, backed by a large language model, offer radically new programming and operating system paradigms.
  arXiv  Detail & Related papers  (2024-08-07T17:11:31Z)
• Code-Switched Language Identification is Harder Than You Think [69.63439391717691]
  Code switching is a common phenomenon in written and spoken communication. We look at the application of building CS corpora. We make the task more realistic by scaling it to more languages. We reformulate the task as a sentence-level multi-label tagging problem to make it more tractable.
  arXiv  Detail & Related papers  (2024-02-02T15:38:47Z)
• Can Large Language Models Write Parallel Code? [0.5317767988097261]
  Large language models are increasingly becoming a popular tool for software development. In this paper, we study the capabilities of state-of-the-art language models to generate parallel code.
  arXiv  Detail & Related papers  (2024-01-23T08:25:12Z)
• The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics [74.99898531299148]
  This research examines vocabulary trimming (VT) inspired by restricting embedding entries to the language of interest to bolster time and memory efficiency. We apply two languages to trim the full vocabulary - Unicode-based script filtering and corpus-based selection - to different language families and sizes. It is found that VT reduces the memory usage of small models by nearly 50% and has an upper bound of 25% improvement in generation speed.
  arXiv  Detail & Related papers  (2023-11-16T09:35:50Z)
• Retentive Network: A Successor to Transformer for Large Language Models [91.6652200825638]
  We propose Retentive Network (RetNet) as a foundation architecture for large language models. We theoretically derive the connection between recurrence and attention. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference.
  arXiv  Detail & Related papers  (2023-07-17T16:40:01Z)
• Prompting Is Programming: A Query Language for Large Language Models [5.8010446129208155]
  We present the novel idea of Language Model Programming (LMP) LMP generalizes language model prompting from pure text prompts to an intuitive combination of text prompting and scripting. We show that LMQL can capture a wide range of state-of-the-art prompting methods in an intuitive way.
  arXiv  Detail & Related papers  (2022-12-12T18:09:09Z)
• QParallel: Explicit Parallelism for Programming Quantum Computers [62.10004571940546]
  We present a language extension for parallel quantum programming. QParallel removes ambiguities concerning parallelism in current quantum programming languages. We introduce a tool that guides programmers in the placement of parallel regions by identifying the subroutines that profit most from parallelization.
  arXiv  Detail & Related papers  (2022-10-07T16:35:16Z)
• StreamBlocks: A compiler for heterogeneous dataflow computing (technical report) [1.5293427903448022]
  This work introduces StreamBlocks, an open-source compiler and runtime that uses the CAL dataflow programming language to partition computations across platforms. StreamBlocks supports exploring the design space with a profile-guided tool that helps identify the best hardware-software partitions.
  arXiv  Detail & Related papers  (2021-07-20T08:46:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.