Orla: A Library for Serving LLM-Based Multi-Agent Systems

• URL: http://arxiv.org/abs/2603.13605v1
• Date: Fri, 13 Mar 2026 21:25:29 GMT
• Title: Orla: A Library for Serving LLM-Based Multi-Agent Systems
• Abstract summary: We introduce Orla, a library for constructing and running LLM-based agentic systems.<n>Orla provides a general abstraction that separates request execution from workflow-level policy.<n>We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline.
• Score: 13.749935259370737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Orla, a library for constructing and running LLM-based agentic systems. Modern agentic applications consist of workflows that combine multiple LLM inference steps, tool calls, and heterogeneous infrastructure. Today, developers typically build these systems by manually composing orchestration code with LLM serving engines and tool execution logic. Orla provides a general abstraction that separates request execution from workflow-level policy. It acts as a serving layer above existing LLM inference engines: developers define workflows composed of stages, while Orla manages how those stages are mapped, executed, and coordinated across models and backends. It provides agent-level control through three mechanisms: a stage mapper, which assigns each stage to an appropriate model and backend; a workflow orchestrator, which schedules stages and manages their resources and context; and a memory manager, which manages inference state such as the KV cache across workflow boundaries. We demonstrate Orla with a customer support workflow that exercises many of its capabilities. We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline, while workflow-level cache management reduces time-to-first-token.

Related papers

• FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production [55.71860113185132]
  We introduce FRAMEWORKERS, a task-centric and workspace-grounded multi-agent framework for open-ended video production.<n>A central Director formulates video creation as dynamic task management, continuously editing a Task Stack to determine which subtask to execute next and which sub-agent to invoke.<n>An Assistant serves as the execution layer, grounding each selected task in a shared Workspace, retrieving the required assets and context, invoking the assigned sub-agent, and persisting the resulting artifacts.<n>Experiments show that FRAMEWORKERS outperforms strong planners in routing accuracy, recovers reliably from runtime failures, generalizes to unseen sub
  arXiv  Detail & Related papers  (2026-08-30T14:28:50Z)
• Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective [3.9457939396735573]
  Helium is a workflow-aware serving framework that models agentic workloads as query plans and treats LLM invocations as first-class operators.<n>Helium bridges classic query optimization principles with LLM serving, achieving up to 1.56x speedup over state-of-the-art agent serving systems.
  arXiv  Detail & Related papers  (2026-03-17T04:03:18Z)
• A Declarative Language for Building And Orchestrating LLM-Powered Agent Workflows [0.0]
  We present a declarative system that separates agent workflow specification from implementation.<n>Our results demonstrate 60% reduction in development time, and 3x improvement in deployment velocity compared to imperative implementations.<n>We show that complex involving product search, personalization, and cart management can be expressed in under 50 lines of DSL compared to 500+ lines of imperative code.
  arXiv  Detail & Related papers  (2025-12-22T05:03:37Z)
• Opus: A Prompt Intention Framework for Complex Workflow Generation [0.0]
  Opus Prompt Intention Framework is designed to improve complex Generation with instruction-tuned Large Language Models (LLMs)<n>We present a customizable Intention Capture system to extract Signals and Intentions from user queries.<n>We provide empirical evidence that the proposed system significantly improves Generation quality compared to direct generation from user queries.
  arXiv  Detail & Related papers  (2025-07-15T13:13:07Z)
• LLM-AutoDiff: Auto-Differentiate Any LLM Workflow [58.56731133392544]
  We introduce LLM-AutoDiff: a novel framework for Automatic Prompt Engineering (APE)<n>LLMs-AutoDiff treats each textual input as a trainable parameter and uses a frozen backward engine to generate feedback-akin to textual gradients.<n>It consistently outperforms existing textual gradient baselines in both accuracy and training cost.
  arXiv  Detail & Related papers  (2025-01-28T03:18:48Z)
• WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models [105.46456444315693]
  We presentLLM, a data-centric framework to enhance the capability of large language models in workflow orchestration. It first constructs a large-scale fine-tuningBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. LlamaLlama demonstrates a strong capacity to orchestrate complex APIs, while also achieving notable generalization performance.
  arXiv  Detail & Related papers  (2024-11-08T09:58:02Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• Teola: Towards End-to-End Optimization of LLM-based Applications [13.478509565946354]
  Large language model (LLM)-based applications contribute to the end-to-end latency.<n>Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions.<n>We propose fine-grained end-to-end orchestration, which utilizes task primitives as the basic units and represents each query's workflow as a primitive-level dataflow graph.
  arXiv  Detail & Related papers  (2024-06-29T05:59:53Z)
• Small LLMs Are Weak Tool Learners: A Multi-LLM Agent [73.54562551341454]
  Large Language Model (LLM) agents significantly extend the capabilities of standalone LLMs. We propose a novel approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer. This modular framework facilitates individual updates and the potential use of smaller LLMs for building each capability.
  arXiv  Detail & Related papers  (2024-01-14T16:17:07Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.