HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks

• URL: http://arxiv.org/abs/2604.09937v1
• Date: Fri, 10 Apr 2026 22:33:39 GMT
• Title: HealthAdminBench: Evaluating Computer-Use Agents on Healthcare Administration Tasks
• Abstract summary: Healthcare administration accounts for over $1 trillion in annual spending, making it a promising target for computer-use agents (CUAs)<n>HealthAdminBench is a benchmark comprising four realistic GUI environments: an EHR, two payer portals, and a fax system, and 135 expert-defined tasks spanning three administrative task types.<n>We evaluate seven agent configurations under multiple prompting and observation settings and find that, despite strong subtask performance, end-to-end reliability remains low.
• Score: 19.2699825267806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Healthcare administration accounts for over $1 trillion in annual spending, making it a promising target for LLM-based computer-use agents (CUAs). While clinical applications of LLMs have received significant attention, no benchmark exists for evaluating CUAs on end-to-end administrative workflows. To address this gap, we introduce HealthAdminBench, a benchmark comprising four realistic GUI environments: an EHR, two payer portals, and a fax system, and 135 expert-defined tasks spanning three administrative task types: Prior Authorization, Appeals and Denials Management, and Durable Medical Equipment (DME) Order Processing. Each task is decomposed into fine-grained, verifiable subtasks, yielding 1,698 evaluation points. We evaluate seven agent configurations under multiple prompting and observation settings and find that, despite strong subtask performance, end-to-end reliability remains low: the best-performing agent (Claude Opus 4.6 CUA) achieves only 36.3 percent task success, while GPT-5.4 CUA attains the highest subtask success rate (82.8 percent). These results reveal a substantial gap between current agent capabilities and the demands of real-world administrative workflows. HealthAdminBench provides a rigorous foundation for evaluating progress toward safe and reliable automation of healthcare administrative workflows.

Related papers

• RobustSGPO: Search-Space Control for Agent Harness Evolution [56.722805974785125]
  We introduce RobustSGPO, which specifies the requested edit, constructs and checks the patch, and continues search from either the incumbent or retained snapshots.<n>We evaluate permission scheduling, cumulative controls, and task-family transfer in the AgentX brainstorming workflow using 120 tasks, 95 runs, and 7,350 candidate attempts.
  arXiv  Detail & Related papers  (2026-09-09T03:00:03Z)
• Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam? [90.39633603556767]
  Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested.<n>To quantify this capability, we introduce an evaluation based on China's National Computer Rank Examination (NCRE)<n>We benchmark 7 frontier LLMs and observe stark limitations: single-turn models score a maximum of 36.6%.<n>A stronger agentic system with execution feedback, iterative repair, and broader Office automation access reaches 68.8%, but remains below the 95.5% community-reference score used as a scoring sanity check.
  arXiv  Detail & Related papers  (2026-06-09T14:59:14Z)
• AutoMedBench: Towards Medical AutoResearch with Agentic AI Models [31.335309699875264]
  AutoMedBench is a workflow-aware benchmark for autonomous medical-AI research.<n>It comprises long-horizon tasks with each run averaging 33 agent turns.
  arXiv  Detail & Related papers  (2026-06-01T09:22:55Z)
• MetaCogAgent: A Metacognitive Multi-Agent LLM Framework with Self-Aware Task Delegation [11.486913474115402]
  We propose MetaCogAgent, a multi-agent large language model (LLM) framework where each agent is equipped with a Metacognitive Self-Assessment Unit.<n>We show that MetaCogAgent achieves 82.4% task accuracy while using 5% fewer API calls than AutoGen and 34% fewer than ensemble voting.
  arXiv  Detail & Related papers  (2026-05-17T07:12:04Z)
• CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? [150.84850629123287]
  End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks.<n>$$-Bench is a benchmark of long-horizon healthcare across three domains: provider prior authorization, payer utilization management, and care management.<n>Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass3, and executing all tasks in a single session slumps the performance to 3.8%.
  arXiv  Detail & Related papers  (2026-05-15T22:34:31Z)
• EnactToM: An Evolving Benchmark for Functional Theory of Mind in Embodied Agents [75.01735520608075]
  Existing benchmarks mostly test literal Theory of Mind (ToM) by asking direct belief questions.<n>We introduce EnactToM, an evolving benchmark of 300 embodied multi-agent tasks set in a 3D household.<n>On the hard split, all seven evaluated frontier models score 0.0% Pass3 on functional task completion, while averaging 45.0% on literal belief probes.
  arXiv  Detail & Related papers  (2026-05-11T00:04:19Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• ART: Action-based Reasoning Task Benchmarking for Medical AI Agents [0.0]
  We introduce Action-based Reasoning clinical Task benchmark for medical AI agents.<n>We identify three dominant error categories: retrieval failures, aggregation errors, and conditional logic misjudgments.<n>Our four-stage pipeline produces diverse, clinically validated tasks grounded in real patient data.
  arXiv  Detail & Related papers  (2026-01-13T21:26:11Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability [35.18016233072556]
  We present UI-CUBE (UiPath Computer Use BEnchmark), a benchmark comprising 226 tasks across two difficulty tiers.<n>Our evaluation covers simple UI interactions (136 tasks) and complex including copy-paste tasks (50 tasks) and enterprise application scenarios (40 tasks)<n>We find that while current CUAs can manipulate individual interface elements, they cannot yet function as reliable workflow automation tools.
  arXiv  Detail & Related papers  (2025-11-21T10:47:22Z)
• Randomized Controlled Trials for Conditional Access Optimization Agent [0.9558392439655014]
  We report results from the first randomized controlled trial (RCT) evaluating an AI agent for Conditional Access (CA) policy management in Microsoft Entra.<n>Agent access produced substantial gains: accuracy improved by 48% and task completion time decreased by 43% while holding accuracy constant.<n>These findings demonstrate that purpose-built AI agents can significantly enhance both speed and accuracy in identity administration.
  arXiv  Detail & Related papers  (2025-11-17T19:33:03Z)
• CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent [46.41047559759938]
  Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces.<n> Reward models offer promising alternatives, but their effectiveness on CUA evaluation remains largely underexplored.<n>We present CUARewardBench, comprising four key contributions.
  arXiv  Detail & Related papers  (2025-10-21T12:53:40Z)
• SCUBA: Salesforce Computer Use Benchmark [63.66753028386581]
  SCUBA is a benchmark designed to evaluate computer-use agents on customer relationship management ( CRM) within the Salesforce platform.<n> SCUBA contains 300 task instances derived from real user interviews, spanning three primary personas, platform administrators, sales representatives, and service agents.<n>We benchmark a diverse set of agents under both zero-shot and demonstration-augmented settings.
  arXiv  Detail & Related papers  (2025-09-30T16:48:49Z)
• GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging [41.754784344572286]
  We release GitTaskBench, a benchmark for evaluating code agents in real-world scenarios.<n>Each task pairs a relevant repository with an automated, human-curated evaluation harness.<n>We also propose the alpha-value metric to quantify the economic benefit of agent performance.
  arXiv  Detail & Related papers  (2025-08-26T12:48:05Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.