CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents

• URL: http://arxiv.org/abs/2606.22000v1
• Date: Sat, 20 Jun 2026 11:34:52 GMT
• Title: CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents
• Abstract summary: CFAgentBench is a benchmark for autonomous construction-finance agents.<n>It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families.<n>The strongest agent reaches pass1 = 0.67 but only pass5 = 0.38.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce CFAgentBench, a reproducible, self-hostable environment and benchmark for autonomous construction-finance agents: a CFO/controller-class agent operating across the real software stack a US construction finance team runs - ERP, project management, email, documents, pay applications, payroll, certified payroll, lien waivers, and bank/treasury portals. It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families, every family grounded in a real source; a self-validated subset of 40 tasks (54 with a project-management extension) is compiled into oracle-validated executable evaluators, the runnable suite reported here. Following WebArena, the benchmark runs on an executable environment rather than static traces: 35 mock applications (31 reconciled to one company book, plus 4 PM platforms) over 9 archetypes, each implementing a uniform self-hostable app contract, so every task is graded by functional correctness - a state diff plus forbidden-side-effect checks plus required-output regexes - with an LLM judge used only for reply quality, never as reward. A distinguishing principle is a money-movement guard: 278 instances embed a payment, payroll, e-signature, or e-filing step where the correct behavior is to stop and stage for human approval, and executing even the correct transaction fails the task. The public split (n=711) is sized for a 95% Wilson half-width of +/-4.1%; a private, contamination-protected split (n=303) is reserved for remote scoring. In a first three-model open-weight sweep (k=5), the strongest agent reaches pass^1 = 0.67 but only pass^5 = 0.38 - losing 43% of its successes when required to repeat them under temperature-0 decoding. The within-model pass^1 to pass^5 collapse and sharp per-domain heterogeneity are clear evidence that single-attempt accuracy overstates deployable construction-finance competence.

Related papers

• From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix [55.08143827481755]
  We consolidate traffic from over 200 internal applications onto a single model.<n>We train a separate GRPO expert per axis and merge them via two-stage SLERP.<n>The model absorbs 50% of platform traffic, 116M requests per month, at a fraction of the serving cost.
  arXiv  Detail & Related papers  (2026-09-01T17:39:26Z)
• PACE: Policy-Attested Contract Execution for Safe AI Agents in Decentralized Finance [9.22772901752634]
  We present PACE, a transaction-level authorization framework that interposes between an LLM-based agent and on-chain execution.<n> PACE cryptographically bind the approved intent, policy, and simulation report to the exact execution bytes, with replay and expiration protection.<n> PACE achieves a 0.00 unsafe execution rate and 0.00 false-positive rate on benign tasks, compared to 0.80 for the unguarded baseline.
  arXiv  Detail & Related papers  (2026-08-18T00:22:46Z)
• Test-time reasoning effort and unauthorized tool use in language-model agents: a prespecified equivalence study [1.6787220460106658]
  Language-model agents that execute through tool calls operate under access-control policies that restrict which operations each role may perform.<n>We vary reasoning effort (low, max) inside GPT-5.6 across 14 confirmatory scenarios of TRIO-20.<n>Across 840 trajectories and two model tiers, no unauthorized tool call occurred.
  arXiv  Detail & Related papers  (2026-08-04T06:01:14Z)
• AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints [0.0]
  AgentSLABench is a resource-aware evaluation framework for autonomous AI agents.<n>It measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets.
  arXiv  Detail & Related papers  (2026-08-01T18:17:47Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows [8.909548943361637]
  BenchAgent places single-agent, fixed multi-agent (MAS) and evolving MAS under one normalized execution and logging protocol.<n> BenchAgent evaluates these substrate-internal across ten reasoning, coding, and tool-use benchmarks with GPT-4.1.<n>On the PAE GAIA snapshot, a Claude-Code-style runtime workflow reaches 66.72% overall and 69.23% on Level 3.
  arXiv  Detail & Related papers  (2026-06-04T03:50:47Z)
• Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents, with an Affine-Typed Rust Mitigation as a Case Study [0.0]
  We build a Rust crate that operationalizes affine ownership so that cloning, double-spending, or using a budget after delegating it are compile errors rather than runtime hazards an operator must remember to avoid.<n>Across five runtimes, three providers, and a temperature-stratified live-API test, the approach reports zero cap violations and zero false refusals, at operational parity with concurrent work.
  arXiv  Detail & Related papers  (2026-06-02T10:46:57Z)
• CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? [150.84850629123287]
  End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks.<n>$$-Bench is a benchmark of long-horizon healthcare across three domains: provider prior authorization, payer utilization management, and care management.<n>Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass3, and executing all tasks in a single session slumps the performance to 3.8%.
  arXiv  Detail & Related papers  (2026-05-15T22:34:31Z)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox [61.862814740220806]
  $textbfComplexMCP$ is a benchmark designed to evaluate agents in rigorous conditions.<n>Built on the Model Context Protocol (MCP), $textbfComplexMCP$ provides over 300 meticulously tested tools.
  arXiv  Detail & Related papers  (2026-05-11T16:20:51Z)
• Towards Optimal Agentic Architectures for Offensive Security Tasks [3.93181912653522]
  We introduce a benchmark of 20 interactive targets (10 web/API and 10 binary)<n>Each exposing one-reachable ground-truth vulnerability, evaluated in whitebox and blackbox modes.<n> MAS-Indep attains the highest validated detection rate (64.2%), while SAS is the strongest efficiency baseline at $0.058 per validated finding.
  arXiv  Detail & Related papers  (2026-04-20T18:17:51Z)
• HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads [0.0]
  In a motivating incident, 3 of 11 parallel agents died from connection resets and HTTP 502 errors - a 27% failure rate.<n>We present HIVEMIND, a transparent HTTP proxy that applies five OS-inspired scheduling primitives to eliminate the failure modes caused by uncoordinated parallel execution.
  arXiv  Detail & Related papers  (2026-04-18T18:59:33Z)
• Capable but Unreliable: Canonical Path Deviation as a Causal Mechanism of Agent Failure in Long-Horizon Tasks [0.38991526486631006]
  We argue that many reliability failures are caused by drift from a task's latent solution structure, not capability failures.<n>We establish this causally using a natural experiment that holds model capability and task difficulty fixed by construction.
  arXiv  Detail & Related papers  (2026-02-22T02:37:57Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.