CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?

• URL: http://arxiv.org/abs/2605.16679v2
• Date: Tue, 19 May 2026 05:51:20 GMT
• Title: CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?
• Abstract summary: End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks.<n>$$-Bench is a benchmark of long-horizon healthcare across three domains: provider prior authorization, payer utilization management, and care management.<n>Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass3, and executing all tasks in a single session slumps the performance to 3.8%.
• Score: 150.84850629123287
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks: policy density, decisions must be grounded in a large library of medical, insurance, and operational rules; Multi-role composition: a single task requires the agent to play multiple roles with handoffs; and multilateral interaction: intermediate workflow steps are multi-turn dialogs, such as peer-to-peer review and patient outreach. We introduce $χ$-Bench, a benchmark of long-horizon healthcare workflows across three domains: provider prior authorization, payer utilization management, and care management. Each task hands the agent a clinical case in a high-fidelity simulator of 20 healthcare apps exposed via 87 MCP tools, which it must drive to a terminal status through tool calls and writing the role's artifacts, guided by a 1,290+ document managed-care operations handbook skill. Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass^3, and executing all tasks in a single session slumps the performance to 3.8%. These results raise the hypothesis that similar gaps are likely to surface in other policy-dense, role-composed, irreversible enterprise domains.

Related papers

• READY or Not: Reliable Enterprise Agent Deployment [16.430548759635037]
  READY is a framework for qualifying AI agents for deployment on enterprise.<n>It measures the reliability and operating cost of the human-AI system.<n>It statistically qualifies it on held-out cases.
  arXiv  Detail & Related papers  (2026-09-02T04:34:48Z)
• DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows [69.12339622053588]
  We introduce DuMateBench, a real-session benchmark reconstructed from anonymized and privacy-screened user sessions.<n>The resulting benchmark comprises 200 tasks spanning 8 broad scenarios and 17 fine-grained capability categories.<n>We execute these tasks in isolated Docker containers injected with three forms of real-world environmental complexity: Insufficient, Unstable, and Noisy.
  arXiv  Detail & Related papers  (2026-08-27T02:36:23Z)
• ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents [61.184954205350984]
  ClawArena-Team is a benchmark of 41 multi-turn, multimodal, multi-agent scenarios spanning 258 evaluation rounds and 72 staged updates.<n>All scoring is execution-based with no LLM judge: an overall score -- the Subagent-Management Score (SMS) -- multiplies task correctness by a least-privilege and modality-routing factor.
  arXiv  Detail & Related papers  (2026-06-30T06:06:08Z)
• CFAgentBench: A Reproducible Environment and Benchmark for Autonomous Construction-Finance Agents [0.0]
  CFAgentBench is a benchmark for autonomous construction-finance agents.<n>It contains 1,014 machine-gradeable task specifications across 8 domains and 77 families.<n>The strongest agent reaches pass1 = 0.67 but only pass5 = 0.38.
  arXiv  Detail & Related papers  (2026-06-20T11:34:52Z)
• MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation [11.486913474115402]
  We propose MetaCogAgent, a multi-agent large language model (LLM) framework where each agent is equipped with a Metacognitive Self-Assessment Unit.<n>We show that MetaCogAgent achieves 82.4% task accuracy while using 5% fewer API calls than AutoGen and 34% fewer than ensemble voting.
  arXiv  Detail & Related papers  (2026-05-17T07:12:04Z)
• HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks [19.2699825267806]
  Healthcare administration accounts for over $1 trillion in annual spending, making it a promising target for computer-use agents (CUAs)<n>HealthAdminBench is a benchmark comprising four realistic GUI environments: an EHR, two payer portals, and a fax system, and 135 expert-defined tasks spanning three administrative task types.<n>We evaluate seven agent configurations under multiple prompting and observation settings and find that, despite strong subtask performance, end-to-end reliability remains low.
  arXiv  Detail & Related papers  (2026-04-10T22:33:39Z)
• Agentic-MME: What Agentic Capability Really Brings to Multimodal Intelligence? [35.30497528897595]
  Agentic-MME is a process-verified benchmark for Multimodal Agentic Capabilities.<n>It contains 418 real-world tasks across 6 domains and 3 difficulty levels.<n>It features over 2,000 stepwise checkpoints that average 10+ person-hours of manual annotation per task.
  arXiv  Detail & Related papers  (2026-04-03T13:02:01Z)
• ALARA for Agents: Least-Privilege Context Engineering Through Portable Composable Multi-Agent Teams [0.0]
  We introduce a declarative context-agent-tool (CAT) data layer expressed through interrelated files that scope each agent's tool access and context to the minimum its role requires.<n>Because the system parses and enforces these files structurally, modifying an agent's tool list produces a guaranteed behavioral change rather than a suggestion the model may or may not follow.<n>We evaluate 22 locally-hosted models from 0.6B to 35B parameters across 115 practical tasks spanning file operations, web search, multi-step scripting, tool chaining, and multi-agent delegation.
  arXiv  Detail & Related papers  (2026-03-20T18:00:09Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation [89.24729958546168]
  Smartphone agents are increasingly important for helping users control devices efficiently.<n>We present SPA-Bench, a comprehensive SmartPhone Agent Benchmark designed to evaluate (M)LLM-based agents.
  arXiv  Detail & Related papers  (2024-10-19T17:28:48Z)
• Policy Diagnosis via Measuring Role Diversity in Cooperative Multi-agent RL [107.58821842920393]
  We quantify the agent's behavior difference and build its relationship with the policy performance via bf Role Diversity We find that the error bound in MARL can be decomposed into three parts that have a strong relation to the role diversity. The decomposed factors can significantly impact policy optimization on three popular directions.
  arXiv  Detail & Related papers  (2022-06-01T04:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.