APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows

• URL: http://arxiv.org/abs/2608.29128v1
• Date: Sat, 29 Aug 2026 08:12:11 GMT
• Title: APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows
• Abstract summary: APIFlow-Bench is a fully auditable benchmark for long-horizon dependent REST-API performance.<n>We generate synthetic API worlds forward, subtask by subtask.<n>We release all answer keys and 44,362 unredacted execution transcripts.
• Score: 23.621239974604205
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tool-using agents are commonly evaluated by a single bit: whether an end-to-end workflow completed. This metric fails to distinguish failures that matter in production, such as expired credentials, malformed payloads, or correct execution followed by incorrect final delivery. We introduce APIFlow-Bench, a fully auditable benchmark for long-horizon, dependent REST-API workflows that decomposes performance into seven engineering capabilities and requires agents to produce answers supported by the actual call path. We generate synthetic API worlds forward, subtask by subtask; each subtask is admitted only after a zero-LLM self-test triad verifies its grader and an oracle establishes solvability, and an adversarial audit identified and fixed six grader exploits. Grading is deterministic and provenance-sensitive: a state check traces a mock-minted canary through the API data flow to the response the answer must originate from, and a typed answer card is verified field by field. We release all answer keys and 44,362 unredacted execution transcripts. Across 19 frontier and open-weight models under one neutral scaffold, we find: (1) longer dependency chains degrade success, from 93% on individual subtasks to 74% on clean 20-subtask chains and 61% when including the 8% of chain trials that a model-consensus screen flags as passed by no model; (2) reliability separates models more than best-case capability, with best-of-five spanning seven points but all-five-of-five reliability spanning 44 points; (3) the independent-error account of compounding failure does not fit the data: pass rates on 20-subtask chains are 33 percentage points above the product of subtask-level rates, and on the clean slice 77% of failing runs reached the correct final state and failed only at delivery.

Related papers

• SkillGLoW: Procedural-Family Skill Consolidation for Self-Improving Agents on Long-Horizon Task Streams [55.90739575113059]
  We argue the missing unit of reuse is the solving procedure shared by a cluster of related tasks, and build SkillGLoW around it.<n>Across four benchmarks (mathematical reasoning, terminal automation, software repair, and embodied control) and three models, the priors gain 17.2 points (hard) over the no-skill baseline on average.
  arXiv  Detail & Related papers  (2026-09-02T07:31:18Z)
• UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations [68.29650478987561]
  UI-Mate is a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning.<n>It sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld and 66.2% on WindowsAgentArena.
  arXiv  Detail & Related papers  (2026-08-16T20:59:59Z)
• Business Truth, not SQL Accuracy: A Rule-Gated 7B Analytics Agent Outperforms a Direct-Prompted 32B Baseline [0.0]
  This paper introduces WarehouseBench, 400 frozen tasks over two synthetic warehouses in which roughly half the correct responses are a clarification, an abstention or a refusal.<n>QueryProof, a 7B agent, uses rules derived from a semantic layer and physical catalog to determine its behaviour.
  arXiv  Detail & Related papers  (2026-08-10T08:14:24Z)
• Turning Interaction History into Execution State: A Runtime Layer for Long-Horizon Coding Agents [48.967927222079965]
  Ledger is a runtime layer that distills an agent's completed interactions into an explicit execution state.<n> Ledger keeps this state in an online execution ledger and applies it at two boundaries of every step.<n>Across all 500 SWE-bench Verified instances, Ledger raises Pass@1 from 56.2% to 64.2% with GPT-5 mini and from 75.8% to 81.0% with MiniMax M2.5.
  arXiv  Detail & Related papers  (2026-08-01T18:25:00Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications [0.0]
  RuBench 1.0 is a benchmark of 25 tasks mined from recent fix commits in five live open-source repositories.<n>Each task is specified in Russian, written from scratch in the style of an actual customer request, not translated.<n>All 25 fix commits postdate the training-data cutoffs of every evaluated model.
  arXiv  Detail & Related papers  (2026-07-07T15:41:22Z)
• CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents [0.0]
  CFAgentBench is a benchmark for autonomous construction-finance agents.<n>It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families.<n>The strongest agent reaches pass1 = 0.67 but only pass5 = 0.38.
  arXiv  Detail & Related papers  (2026-06-20T11:34:52Z)
• SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows [16.693609667845948]
  We introduce SKILL.nb, a framework for governing reusable agent with evidence-versus-calibrated lifecycle policies.<n>SKILL.nb uses selective formalization: execution decides which workflow steps should become executable code.<n> gate-conditioned execution lets each step run code when its gates validate, or fall back locally when drift invalidates the executable realization.
  arXiv  Detail & Related papers  (2026-06-06T08:27:18Z)
• ContractBench: Can LLM Agents Preserve Observation Contracts? [9.057486468322933]
  We show that observation contract compliance is an emergent, regression-prone capability.<n>We introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two failure modes.<n>We report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%; (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B; (iii) non-monotonic scaling across the GPT-5 family: agentic post-training can erode
  arXiv  Detail & Related papers  (2026-05-17T06:37:04Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.