Substrate-Portable Execution for Production LLM Workflows

• URL: http://arxiv.org/abs/2609.06128v1
• Date: Sat, 05 Sep 2026 14:52:35 GMT
• Title: Substrate-Portable Execution for Production LLM Workflows
• Abstract summary: Execution semantics are often coupled to one runtime.<n>Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads.<n>We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Production LLM agents execute tool-calling loops, retrieval chains, and compositional workflows in multiple modes, yet execution semantics are often coupled to one runtime. We encountered this portability problem in Rufus, a conversational AI assistant with a large tool catalog that serves millions of Amazon customers. Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads such as evaluation and content pregeneration. Each mode has distinct service-level objectives and typically uses a separate runtime. Reusing streaming orchestration makes asynchronous and batch workloads blocking and prevents use of batch inference APIs, which offer a 50 percent discount at published prices. We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate. Developers define a workflow once as a typed dataflow graph. The platform compiles the graph to in-process streaming for real-time serving, durable AWS SWF orchestration for asynchronous execution, or distributed Apache Flink stream processing for batch inference. No workflow code changes are required. LLM inference is represented as a suspendable graph node whose behavior depends on the substrate: streaming delivery online, durable retry asynchronously, and batched submission offline. We validated dozens of production agent configurations across five orchestration patterns: single-inference RAG, iterative ReAct, compositional PreAct, conditional routing, and multi-agent deep research. Across all three bindings, we found no detectable difference in output quality. Batch execution reduced per-query inference cost in line with published batch API pricing while operating alongside the streaming path at production scale.

Related papers

• Decoupling Readiness from Release for Tail-Aware Scheduling of Agentic LLM Workflows [25.368121854431035]
  tail-risk-aware turn release scheduling method that jointly decides which ready turn to release next.<n>We present a tail-risk-aware turn release scheduling method that jointly decides which ready turn to release next.
  arXiv  Detail & Related papers  (2026-09-10T01:35:10Z)
• FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production [55.71860113185132]
  We introduce FRAMEWORKERS, a task-centric and workspace-grounded multi-agent framework for open-ended video production.<n>A central Director formulates video creation as dynamic task management, continuously editing a Task Stack to determine which subtask to execute next and which sub-agent to invoke.<n>An Assistant serves as the execution layer, grounding each selected task in a shared Workspace, retrieving the required assets and context, invoking the assigned sub-agent, and persisting the resulting artifacts.<n>Experiments show that FRAMEWORKERS outperforms strong planners in routing accuracy, recovers reliably from runtime failures, generalizes to unseen sub
  arXiv  Detail & Related papers  (2026-08-30T14:28:50Z)
• AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications [0.0]
  This paper presents AiFlow, a tokennative reactive orchestration model that normalizes provider deltas into typed ContextT> events.<n>AiFlow does not alter provider-side TTFT but reduces Application TTFPT by 70.9-94.7% versus aggregation and keeps runtime-owned queue depth within declared bounds.
  arXiv  Detail & Related papers  (2026-08-01T09:33:29Z)
• A Readiness-Driven Runtime for Pipeline-Parallel Training under Runtime Variability [12.676135710714812]
  Pipeline is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication.<n>We present-Readiness-First Pipeline (R RFP), a readiness-driven runtime-parallel training.<n>R RFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint order for ranking currently ready work.
  arXiv  Detail & Related papers  (2026-05-18T17:59:18Z)
• Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines [0.7551404527457958]
  We describe a new agentic serving system that efficiently schedules arbitrary multi-LLM agentic onto a GPU cluster.<n>Scepsy achieves up to 2.4x higher throughput and 27x lower latency compared to systems that optimize LLMs independently.
  arXiv  Detail & Related papers  (2026-04-16T16:15:29Z)
• Orla: A Library for Serving LLM-Based Multi-Agent Systems [13.749935259370737]
  We introduce Orla, a library for constructing and running LLM-based agentic systems.<n>Orla provides a general abstraction that separates request execution from workflow-level policy.<n>We evaluate Orla on two datasets, showing that stage mapping improves latency and cost compared to a single-model vLLM baseline.
  arXiv  Detail & Related papers  (2026-03-13T21:25:29Z)
• Learning to Compose for Cross-domain Agentic Workflow Generation [56.630382886594184]
  We create an open-source LLM for cross-domain workflow generation.<n>We learn a compact set of reusable workflow capabilities across diverse domains.<n>Our 1-pass generator surpasses SOTA refinement baselines that consume 20 iterations.
  arXiv  Detail & Related papers  (2026-02-11T18:27:22Z)
• FlowSteer: Interactive Agentic Workflow Orchestration via End-to-End Reinforcement Learning [49.369614288007334]
  FlowSteer is an end-to-end reinforcement learning framework that takes a lightweight policy model as the agent and an executable canvas environment.<n>We show that FlowSteer significantly outperforms baselines across various tasks.
  arXiv  Detail & Related papers  (2026-02-02T05:30:42Z)
• Accelerating Language Model Workflows with Prompt Choreography [15.03063157222079]
  We introduce Prompt Choreography, a framework that efficiently executes LLM by maintaining a dynamic, global KV cache.<n>Each LLM call can attend to an arbitrary, reordered subset of previously encoded messages.<n>Prompt Choreography significantly reduces per-message latency.
  arXiv  Detail & Related papers  (2025-12-28T19:21:11Z)
• DyFlow: Dynamic Workflow Framework for Agentic Reasoning [79.19799197382478]
  DyFlow is a dynamic workflow generation framework that adaptively constructs and adjusts reasoning procedures based on task requirements and real-time intermediate feedback.<n>We systematically evaluate DyFlow across diverse domains, including social reasoning, biomedical tasks, mathematical problem solving, and code generation.<n>Results demonstrate that DyFlow significantly outperforms existing baselines, achieving substantial Pass@k improvements and exhibiting robust generalization across diverse domains.
  arXiv  Detail & Related papers  (2025-09-30T10:36:23Z)
• WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models [105.46456444315693]
  We presentLLM, a data-centric framework to enhance the capability of large language models in workflow orchestration. It first constructs a large-scale fine-tuningBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. LlamaLlama demonstrates a strong capacity to orchestrate complex APIs, while also achieving notable generalization performance.
  arXiv  Detail & Related papers  (2024-11-08T09:58:02Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.