AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications

• URL: http://arxiv.org/abs/2608.00558v1
• Date: Sat, 01 Aug 2026 09:33:29 GMT
• Title: AiFlow: Token-Native Reactive Orchestration with Bounded Backpressure for Streaming LLM Applications
• Abstract summary: This paper presents AiFlow, a tokennative reactive orchestration model that normalizes provider deltas into typed ContextT> events.<n>AiFlow does not alter provider-side TTFT but reduces Application TTFPT by 70.9-94.7% versus aggregation and keeps runtime-owned queue depth within declared bounds.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) applications increasingly operate as streaming workflows combining retrieval, tool calls, safety filters, and multi-agent coordination. Although contemporary frameworks expose provider deltas, workflow nodes often treat generation as coarse request-response steps, leaving queue management, worker allocation, ordering, and backpressure to ad hoc callback code. This paper presents AiFlow, a token-native reactive orchestration model that normalizes provider deltas into typed Context<T> events propagated through a directed streaming graph. Each node is managed by a Node Guardian that declares and enforces local queue bounds, worker concurrency, ordering, overflow policy, cancellation propagation, and retry discipline. We formalize the bounded-memory property, present the compilation from a compact DSL and JSON graph form, and provide static validation for type safety, state concurrency, and injection compatibility. Controlled microbenchmarks, captured DeepSeek trace replay (30 runs), descriptive online runs, LangGraph baselines, a streaming RAG workload, and an Ollama local-backend check show that AiFlow does not alter provider-side Model TTFT but reduces Application TTFPT by 70.9-94.7\% versus aggregation and keeps runtime-owned queue depth within declared bounds (93.7-96.5\% MaxQ reduction versus unbounded policies). The supplementary artifact contains scripts, raw traces, machine-readable tables, checksums, and an API-free smoke test; the public implementation is available through the FIT Framework repository.

Related papers

• Substrate-Portable Execution for Production LLM Workflows [0.0]
  Execution semantics are often coupled to one runtime.<n>Rufus supports real-time serving, asynchronous background tasks, and high-volume batch workloads.<n>We present a binding-adaptive agent execution platform that separates workflow definition from execution substrate.
  arXiv  Detail & Related papers  (2026-09-05T14:52:35Z)
• Inference-Time Graph Engineering for Multi-Agent LLM Workflows [51.56038180639833]
  We introduce ReActNet, a training-free framework that compiles a query and a set of role-specialized agents into a sequence of directed communication graphs.<n>We show that ReActNet consistently improves over fixed-topology and learned-topology baselines while maintaining competitive inference cost.
  arXiv  Detail & Related papers  (2026-09-04T23:32:11Z)
• DataFlow-Harness: A Grounded Code-Agent Platform for Constructing Editable LLM Data Pipelines [9.38656029481013]
  We introduce textscDataFlow-Harness, a platform that guides an LLM agent to construct platform-native directed acyclic graphs.<n>On a 12-task data-engineering benchmark, textscDataFlow-Harness achieves a 93.3% observed end-to-end pass rate.
  arXiv  Detail & Related papers  (2026-07-18T03:30:19Z)
• A Formal Hierarchical Architecture for Agentic Orchestration with Stack-Based Execution and Lazy Discovery [5.106901034158075]
  This paper presents a hierarchical, skill-based architecture for agentic orchestration.<n> Capabilities are organized as a rooted tree where internal nodes make routing decisions and leaf nodes execute deterministic tasks.<n>By replacing global memory with localized stack frames, the architecture prevents outputs from one execution branch from leaking into another.
  arXiv  Detail & Related papers  (2026-07-13T06:15:51Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• DiscreteRTC: Discrete Diffusion Policies are Natural Asynchronous Executors [57.944744187489185]
  We propose DiscreteRTC, which replaces external corrections with native unmasking.<n>DiscreteRTC is simpler to implement with 0 lines of code for async inpainting, faster at inference with only 0.7x computation compared with generating actions from scratch, and better at execution with 50% higher success rate in real-world dynamic pick task compared with flow-based RTC.
  arXiv  Detail & Related papers  (2026-04-27T23:04:03Z)
• GraphBit: A Graph-based Agentic Framework for Non-Linear Agent Orchestration [34.588097323063074]
  GraphBit is an engine-orchestrated framework that defines explicitly and deterministically as a directed acyclic graph (DAG)<n>Unlike prompted orchestration, agents in GraphBit operate as typed functions, while a Rust-based engine governs routing, state transitions, and tool invocation, ensuring auditability.<n>A three-tier memory architecture consisting of ephemeral scratch space, structured state, and external connectors isolates context across stages, preventing cascading context that degrades reasoning in long-running pipelines.
  arXiv  Detail & Related papers  (2026-03-08T18:32:28Z)
• CaveAgent: Transforming LLMs into Stateful Runtime Operators [31.548422546991915]
  We present CaveAgent, a framework that transforms the paradigm from "LLM-as-Text-Generator" to "LLM-as-As-Runtime-Runtime"<n>CaveAgent achieves a 10.5% success rate improvement on retail tasks and reduces total token consumption by 28.4% in multi-turn scenarios.
  arXiv  Detail & Related papers  (2026-01-04T15:32:47Z)
• DyFlow: Dynamic Workflow Framework for Agentic Reasoning [79.19799197382478]
  DyFlow is a dynamic workflow generation framework that adaptively constructs and adjusts reasoning procedures based on task requirements and real-time intermediate feedback.<n>We systematically evaluate DyFlow across diverse domains, including social reasoning, biomedical tasks, mathematical problem solving, and code generation.<n>Results demonstrate that DyFlow significantly outperforms existing baselines, achieving substantial Pass@k improvements and exhibiting robust generalization across diverse domains.
  arXiv  Detail & Related papers  (2025-09-30T10:36:23Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorfBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.<n>We also present WorfEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.<n>We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• Online Joint Fine-tuning of Multi-Agent Flows [12.851745991007169]
  I describe a procedure for online joint fine-tuning of an entire flow inspired by the Learning to Search framework. The approach leverages simulator access to reduce preferences over entire episodes to preferences over individual node outputs. I apply to the multi-hop QA dataset Musique achieving a state-of-the-art result.
  arXiv  Detail & Related papers  (2024-06-06T21:21:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.